If you're using XLSForm to create your survey...
It sounds like you want to implement a cascading select.
To limit the number of selections, you should be able to use a constraint of count-selected(.)<=3 as noted in: How to impose strict upper limit on the number of selected answers to 3 for multiple choice questions?
Your survey worksheet should look something like:
+-------------------------+--------------------+----------------------+--------------------------------+----------------------+------------------------+----------+
| type | name | label | choice_filter | constraint | constraint_message | required |
+-------------------------+--------------------+----------------------+--------------------------------+----------------------+------------------------+----------+
| select_one semesters | semester-selection | Select the semester. | | | | yes |
| select_multiple courses | course-selection | Select the courses. | semester=${semester-selection} | count-selected(.)<=3 | Must select 3 or less. | yes |
+-------------------------+--------------------+----------------------+--------------------------------+----------------------+------------------------+----------+
and your choices worksheet should look something like:
+-----------+------+------------+----------+
| list name | name | label | semester |
+-----------+------+------------+----------+
| semesters | 1 | semester 1 | |
| semesters | 2 | semester 2 | |
| | | | |
| courses | 101 | course 101 | 1 |
| courses | 103 | course 103 | 1 |
| courses | 105 | course 105 | 1 |
| courses | 107 | course 107 | 1 |
| courses | 109 | course 109 | 1 |
| courses | 102 | course 102 | 2 |
| courses | 104 | course 104 | 2 |
| courses | 106 | course 106 | 2 |
| courses | 108 | course 108 | 2 |
| courses | 110 | course 110 | 2 |
+-----------+------+------------+----------+
(note: I didn't test this. If this doesn't answer your question please post the survey form that you've tried already and that is not working.)