and not all are children of Abraham because they are his offspring, but “Through Isaac shall your offspring be named.”  This means that it is not the children of the flesh who are the children of God, but the children of the promise are counted as offspring. (ESV)
The Jews were very proud people. They had much to be proud of. As Paul has made very clear the Jewish heritage included the very profound promises of God. The giving of the law, worship, and the covenants were all part of their wonderful history. The Jews viewed Abraham as their "father." It was Abraham whom God called and made a people of to bring redemption to the world.
Paul is making the point here that just being a physical descendant of Abraham does not at all make you a Jew in the sense of being a child of God. You can be a Jew in the physical sense only, and that does not mean that you are a recipient of the promises of God. The children of the promise are those who belong to God.
There is a very interesting passage of Scripture in John 8. The religious leaders of the Jews, the Pharisees were trying to kill Jesus. They viewed him as a heretic, not as their Messiah. In his confrontation with them they claimed that they had Abraham as their father. Did that mean that they were children of God? Were they children of the promise? Listen to Jesus' answer:
 Why do you not understand what I say? It is because you cannot bear to hear my word.  You are of your father the devil, and your will is to do your father's desires. He was a murderer from the beginning, and does not stand in the truth, because there is no truth in him. When he lies, he speaks out of his own character, for he is a liar and the father of lies. (ESV)
Children of God are the children of the promises of God. Those promises are available for all, first to the Jew, and also to the Gentile. Be sure that if you claim God as your father you are doing the will of the father. Otherwise, you might belong to the one that the Pharisees belonged to.