Aug 04 17:52:18.148758 ip-10-0-128-51 systemd[1]: Starting Kubernetes Kubelet... Aug 04 17:52:18.508093 ip-10-0-128-51 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:18.508093 ip-10-0-128-51 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 17:52:18.508093 ip-10-0-128-51 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:18.508093 ip-10-0-128-51 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 17:52:18.508093 ip-10-0-128-51 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:18.509075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.508946 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 17:52:18.512734 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512720 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.512734 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512733 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512738 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512742 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512744 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512747 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512749 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512752 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512755 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512757 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512759 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512762 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512764 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512767 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512769 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512772 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512774 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512776 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512779 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512781 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.512813 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512784 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512786 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512789 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512791 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512795 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512798 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512802 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512806 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512809 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512816 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512819 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512822 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512825 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512827 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512829 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512832 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512834 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512837 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512839 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.513245 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512841 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512844 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512846 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512849 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512851 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512853 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512856 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512860 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512863 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512866 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512868 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512871 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512873 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512876 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512879 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512882 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512884 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512887 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512889 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512892 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.513709 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512895 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512897 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512900 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512902 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512905 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512908 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512910 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512912 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512915 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512917 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512919 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512922 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512924 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512926 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512929 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512931 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512933 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512935 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512938 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512941 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.514211 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512943 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512946 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512949 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512951 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512954 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512956 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512959 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512963 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512966 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512968 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512971 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512973 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512976 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512978 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512980 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512982 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512985 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512987 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512990 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.512992 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.514745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513677 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513684 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513687 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513689 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513692 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513694 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513697 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513699 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513701 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513704 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513709 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513711 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513714 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513719 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513724 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513727 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513730 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513732 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513735 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.515207 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513738 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513740 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513742 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513745 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513747 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513749 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513752 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513756 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513759 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513761 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513764 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513767 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513769 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513772 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513774 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513777 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513779 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513782 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513784 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513786 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.515688 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513789 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513792 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513794 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513797 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513800 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513802 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513804 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513807 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513810 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513813 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513815 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513817 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513820 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513823 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513825 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513827 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513830 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513832 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513835 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.516160 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513837 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513839 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513842 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513844 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513846 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513849 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513852 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513854 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513856 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513859 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513861 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513863 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513865 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513868 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513870 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513872 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513876 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513878 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513880 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513883 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.516745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513886 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513888 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513891 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513893 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513896 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513898 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513900 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513903 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513905 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513907 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513910 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513912 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513914 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513917 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513919 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513922 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513926 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513928 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513931 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.517229 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513933 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.513935 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514265 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514269 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514272 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514275 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514278 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514280 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514283 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514286 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514288 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514292 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514295 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514298 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514301 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514304 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514308 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514310 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514313 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.517695 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514315 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514317 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514320 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514322 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514324 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514326 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514329 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514331 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514333 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514336 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514338 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514340 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514343 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514346 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514348 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514351 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514353 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514356 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514359 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514361 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.518145 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514364 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514366 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514369 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514371 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514374 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514376 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514378 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514381 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514384 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514387 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514389 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514392 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514394 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514396 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514400 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514403 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514406 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514408 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514411 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.519960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514414 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514416 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514418 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514421 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514423 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514426 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514428 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514431 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514433 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514435 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514438 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514440 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514442 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514444 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514447 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514449 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514451 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514454 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514456 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514460 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.521926 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514463 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514465 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514467 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514470 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514472 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514475 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514478 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514480 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514483 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514485 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514488 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514490 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514492 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514510 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514512 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514515 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514517 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514519 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514523 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514526 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.522397 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514529 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514531 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.514534 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514601 2559 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514607 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514612 2559 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514616 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514620 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514623 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514627 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514630 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514634 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514637 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514640 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514643 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514646 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514650 2559 flags.go:64] FLAG: --cgroup-root="" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514653 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514656 2559 flags.go:64] FLAG: --client-ca-file="" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514658 2559 flags.go:64] FLAG: --cloud-provider="external" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514661 2559 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514666 2559 flags.go:64] FLAG: --cluster-domain="" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514669 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514671 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 17:52:18.522909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514674 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514677 2559 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514681 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514683 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514686 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514689 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514694 2559 flags.go:64] FLAG: --contention-profiling="false" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514697 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514699 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514702 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514705 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514709 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514712 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514714 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514717 2559 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514720 2559 flags.go:64] FLAG: --enable-server="true" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514722 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514726 2559 flags.go:64] FLAG: --event-burst="100" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514729 2559 flags.go:64] FLAG: --event-qps="50" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514733 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514736 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514739 2559 flags.go:64] FLAG: --eviction-hard="" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514742 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514745 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514748 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 17:52:18.523461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514752 2559 flags.go:64] FLAG: --eviction-soft="" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514754 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514757 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514760 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514762 2559 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514765 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514768 2559 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514770 2559 flags.go:64] FLAG: --feature-gates="" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514774 2559 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514776 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514779 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514782 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514785 2559 flags.go:64] FLAG: --healthz-port="10248" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514789 2559 flags.go:64] FLAG: --help="false" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514791 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514794 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514797 2559 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514799 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514803 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514806 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514809 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514812 2559 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514815 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 17:52:18.524100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514817 2559 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514820 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514823 2559 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514827 2559 flags.go:64] FLAG: --kube-reserved="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514829 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514832 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514835 2559 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514837 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514840 2559 flags.go:64] FLAG: --lock-file="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514843 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514846 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514849 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514854 2559 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514856 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514859 2559 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514862 2559 flags.go:64] FLAG: --logging-format="text" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514864 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514867 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514869 2559 flags.go:64] FLAG: --manifest-url="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514872 2559 flags.go:64] FLAG: --manifest-url-header="" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514876 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514879 2559 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514888 2559 flags.go:64] FLAG: --max-pods="110" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514891 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514893 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 17:52:18.528330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514896 2559 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514899 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514901 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514904 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514907 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514913 2559 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514916 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514919 2559 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514922 2559 flags.go:64] FLAG: --pod-cidr="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514924 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514935 2559 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514938 2559 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514941 2559 flags.go:64] FLAG: --pods-per-core="0" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514944 2559 flags.go:64] FLAG: --port="10250" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514946 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514949 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03a02399f306e6d11" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514952 2559 flags.go:64] FLAG: --qos-reserved="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514955 2559 flags.go:64] FLAG: --read-only-port="10255" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514958 2559 flags.go:64] FLAG: --register-node="true" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514961 2559 flags.go:64] FLAG: --register-with-taints="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514964 2559 flags.go:64] FLAG: --registry-burst="10" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514966 2559 flags.go:64] FLAG: --registry-qps="5" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514969 2559 flags.go:64] FLAG: --reserved-cpus="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514972 2559 flags.go:64] FLAG: --reserved-memory="" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514975 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 17:52:18.528979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514978 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514980 2559 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514983 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514985 2559 flags.go:64] FLAG: --runonce="false" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514989 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514992 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514995 2559 flags.go:64] FLAG: --seccomp-default="false" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.514997 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515000 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515003 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515010 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515013 2559 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515015 2559 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515018 2559 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515020 2559 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515023 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515026 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515028 2559 flags.go:64] FLAG: --system-cgroups="" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515032 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515037 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515040 2559 flags.go:64] FLAG: --tls-cert-file="" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515042 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515046 2559 flags.go:64] FLAG: --tls-min-version="" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515049 2559 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515052 2559 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 17:52:18.529593 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515054 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515058 2559 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515061 2559 flags.go:64] FLAG: --v="2" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515064 2559 flags.go:64] FLAG: --version="false" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515068 2559 flags.go:64] FLAG: --vmodule="" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515071 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.515074 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515178 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515182 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515184 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515187 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515191 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515194 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515197 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515200 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515203 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515206 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515209 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515212 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515215 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.530187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515219 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515222 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515225 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515228 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515230 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515234 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515236 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515239 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515241 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515244 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515246 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515248 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515250 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515253 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515256 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515258 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515261 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515263 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515265 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515268 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.530685 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515270 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515272 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515274 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515278 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515280 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515283 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515285 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515287 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515289 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515292 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515294 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515296 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515299 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515302 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515304 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515307 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515309 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515312 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515314 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.531159 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515316 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515319 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515321 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515324 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515327 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515329 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515332 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515334 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515337 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515340 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515342 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515345 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515347 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515349 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515352 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515354 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515358 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515361 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515363 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515365 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.531719 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515367 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515370 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515372 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515375 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515377 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515380 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515382 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515384 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515387 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515389 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515392 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515394 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515397 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515400 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515404 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515406 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515409 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515411 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515413 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515416 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.532188 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515418 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515421 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515423 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515425 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515428 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515430 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.515432 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.516032 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.531273 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.531286 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.531306 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.531314 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531373 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531378 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531381 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.532718 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531384 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531387 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531390 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531393 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531396 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531398 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531400 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531403 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531405 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531407 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531410 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531414 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531417 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531420 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531422 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531424 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531427 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531429 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531431 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.533089 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531434 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531436 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531438 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531441 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531443 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531445 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531449 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531452 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531455 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531457 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531460 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531462 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531465 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531467 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531469 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531472 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531474 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531477 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531479 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531482 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.533553 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531484 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531486 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531489 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531491 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531493 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531510 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531512 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531515 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531518 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531520 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531523 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531527 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531529 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531532 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531535 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531537 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531539 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531542 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531545 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531548 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.534020 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531551 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531554 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531558 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531561 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531563 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531566 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531568 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531571 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531574 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531576 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531578 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531581 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531583 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531586 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531588 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531590 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531592 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531595 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531597 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.534521 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531600 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531602 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531604 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531607 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531610 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531612 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531614 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531617 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531619 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531622 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531624 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531627 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531629 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531632 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531636 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531639 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531642 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.534965 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531644 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.531649 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531762 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531767 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531770 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531772 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531775 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531778 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531780 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531783 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531785 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531788 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531790 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531792 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531795 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:18.535374 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531798 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531800 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531804 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531807 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531810 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531813 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531815 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531818 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531820 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531822 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531825 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531827 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531830 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531833 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531835 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531838 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531840 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531843 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531845 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:18.535745 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531847 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531849 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531852 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531854 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531856 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531860 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531863 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531866 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531869 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531871 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531874 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531876 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531879 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531881 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531884 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531887 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531890 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531892 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531895 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:18.536187 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531897 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531899 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531902 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531904 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531906 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531909 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531911 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531914 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531916 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531918 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531921 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531924 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531926 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531928 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531931 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531934 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531936 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531938 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531941 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531943 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:18.536710 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531945 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531948 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531950 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531953 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531955 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531957 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531959 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531962 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531964 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531966 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531969 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531971 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531974 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531976 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531979 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531981 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531983 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531986 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531988 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531990 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:18.537177 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531993 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531995 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.531998 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.532001 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.532003 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.532005 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.532008 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:18.532010 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.532014 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.532114 2559 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.534196 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.535507 2559 server.go:1019] "Starting client certificate rotation" Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.535626 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 17:52:18.537703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.536417 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 17:52:18.538021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.537441 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 17:52:18.541764 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.541749 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 17:52:18.551569 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.551541 2559 log.go:25] "Validated CRI v1 runtime API" Aug 04 17:52:18.554449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.554433 2559 log.go:25] "Validated CRI v1 image API" Aug 04 17:52:18.555259 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.555245 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 17:52:18.559216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.559197 2559 fs.go:135] Filesystem UUIDs: map[3008f21d-b693-4760-8807-b4ceee9c2c7e:/dev/nvme0n1p3 3b1bfa73-a0f2-4b92-85a9-87e217d52127:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Aug 04 17:52:18.559289 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.559215 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 17:52:18.565126 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.565018 2559 manager.go:216] Machine: {Timestamp:2026-08-04 17:52:18.563787367 +0000 UTC m=+0.305941416 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201014 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d359691ff4b360ae965789f6915d4 SystemUUID:ec2d3596-91ff-4b36-0ae9-65789f6915d4 BootID:7d2c54e1-38b1-4d9f-ac85-06af1a3b6cd2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:7e:3a:04:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:7e:3a:04:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:ec:83:db:e2:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 17:52:18.565126 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.565122 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 17:52:18.565230 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.565191 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 17:52:18.567696 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.567676 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 17:52:18.567835 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.567698 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 17:52:18.568418 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.568408 2559 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 17:52:18.568454 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.568421 2559 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 17:52:18.568454 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.568428 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 17:52:18.568454 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.568445 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 17:52:18.569552 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.569543 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 17:52:18.573041 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573029 2559 state_mem.go:36] "Initialized new in-memory state store" Aug 04 17:52:18.573150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573141 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 17:52:18.573190 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573176 2559 kubelet.go:486] "Attempting to sync node with API server" Aug 04 17:52:18.573190 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573190 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 17:52:18.573240 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573203 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 17:52:18.573240 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573220 2559 kubelet.go:398] "Adding apiserver pod source" Aug 04 17:52:18.573240 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573233 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 17:52:18.573581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573554 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 17:52:18.573581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573577 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 17:52:18.573719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.573591 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 17:52:18.575737 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.575720 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 17:52:18.577542 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.577527 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 17:52:18.577595 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.577560 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 17:52:18.578681 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578668 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 17:52:18.578719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578689 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 17:52:18.578719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578695 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 17:52:18.578719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578700 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 17:52:18.578719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578708 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 17:52:18.578719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578714 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578722 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578727 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578734 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578741 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578757 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 17:52:18.578877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.578763 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 17:52:18.579569 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.579560 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 17:52:18.579569 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.579569 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 17:52:18.582659 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.582646 2559 server.go:1268] "Started kubelet" Aug 04 17:52:18.582769 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.582734 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 17:52:18.583546 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.583487 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 17:52:18.583620 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.583568 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 17:52:18.584166 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.584132 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 17:52:18.584253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.584187 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 17:52:18.584299 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.584265 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 17:52:18.584494 ip-10-0-128-51 systemd[1]: Started Kubernetes Kubelet. Aug 04 17:52:18.584979 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.584935 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 17:52:18.585554 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.585541 2559 server.go:310] "Adding debug handlers to kubelet server" Aug 04 17:52:18.590814 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.587443 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-51.ec2.internal.18c8ab9bf5afbab1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-51.ec2.internal,UID:ip-10-0-128-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-51.ec2.internal,},FirstTimestamp:2026-08-04 17:52:18.582616753 +0000 UTC m=+0.324770801,LastTimestamp:2026-08-04 17:52:18.582616753 +0000 UTC m=+0.324770801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-51.ec2.internal,}" Aug 04 17:52:18.591219 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.591202 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 17:52:18.593959 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.593941 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 17:52:18.593959 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.593950 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 17:52:18.594091 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.594056 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:18.594725 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.594708 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 17:52:18.594827 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.594761 2559 factory.go:55] Registering systemd factory Aug 04 17:52:18.594827 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.594775 2559 factory.go:223] Registration of the systemd container factory successfully Aug 04 17:52:18.594938 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.594764 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 17:52:18.594938 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.594889 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 17:52:18.595027 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595014 2559 factory.go:153] Registering CRI-O factory Aug 04 17:52:18.595080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595033 2559 factory.go:223] Registration of the crio container factory successfully Aug 04 17:52:18.595080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595019 2559 reconstruct.go:97] "Volume reconstruction finished" Aug 04 17:52:18.595080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595049 2559 reconciler.go:29] "Reconciler: start to sync state" Aug 04 17:52:18.595204 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595086 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 17:52:18.595204 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595106 2559 factory.go:103] Registering Raw factory Aug 04 17:52:18.595204 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595117 2559 manager.go:1196] Started watching for new ooms in manager Aug 04 17:52:18.595557 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.595547 2559 manager.go:349] Starting recovery of all containers Aug 04 17:52:18.596014 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.595992 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 17:52:18.596138 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.596117 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 17:52:18.600436 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.600413 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m54ch" Aug 04 17:52:18.605443 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.605425 2559 manager.go:354] Recovery completed Aug 04 17:52:18.609346 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.609323 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m54ch" Aug 04 17:52:18.609750 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.609736 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.611646 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.611630 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.611730 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.611662 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.611730 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.611676 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.612193 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612180 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 17:52:18.612193 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612191 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 17:52:18.612300 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612203 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.612300 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612205 2559 state_mem.go:36] "Initialized new in-memory state store" Aug 04 17:52:18.612857 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612841 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.612931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612867 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.612931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.612876 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.613400 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.613387 2559 policy_none.go:49] "None policy: Start" Aug 04 17:52:18.613430 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.613408 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 17:52:18.613430 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.613418 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 17:52:18.614448 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.614345 2559 policy_none.go:47] "Start" Aug 04 17:52:18.653544 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.653530 2559 manager.go:337] "Starting Device Plugin manager" Aug 04 17:52:18.653614 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.653566 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 17:52:18.653614 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.653578 2559 server.go:87] "Starting device plugin registration server" Aug 04 17:52:18.653791 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.653779 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 17:52:18.653833 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.653793 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 17:52:18.653890 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.653875 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 17:52:18.656045 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.654380 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 17:52:18.656045 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.654395 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 17:52:18.656045 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.654804 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 17:52:18.656045 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.654856 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:18.731230 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.731188 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 17:52:18.733288 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.732538 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 17:52:18.733288 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.732569 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 17:52:18.733288 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.732587 2559 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 17:52:18.733288 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.732621 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 17:52:18.734837 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.734819 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:18.754750 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.754720 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.755896 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.755883 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.755957 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.755911 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.755957 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.755923 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.755957 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.755947 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.767262 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.767221 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.767262 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.767242 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-51.ec2.internal\": node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:18.798364 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.798344 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:18.833680 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.833661 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal"] Aug 04 17:52:18.833750 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.833713 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.835286 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.835267 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.835337 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.835298 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.835337 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.835310 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.837447 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.837434 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.837576 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.837552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.837617 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.837589 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.839539 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839526 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.839615 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839536 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.839615 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839552 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.839615 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839563 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.839615 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839575 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.839615 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.839565 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.841728 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.841713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.841807 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.841739 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:18.842410 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.842394 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:18.842485 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.842424 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:18.842485 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.842434 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:18.854987 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.854965 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-51.ec2.internal\" not found" node="ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.858477 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.858463 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-51.ec2.internal\" not found" node="ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.897155 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.897138 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82be4bfc3f07367942d8a0fb59c52b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-51.ec2.internal\" (UID: \"c82be4bfc3f07367942d8a0fb59c52b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.897251 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.897160 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.897251 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.897175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.898998 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.898985 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:18.998247 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82be4bfc3f07367942d8a0fb59c52b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-51.ec2.internal\" (UID: \"c82be4bfc3f07367942d8a0fb59c52b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.998344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.998344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.998344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c82be4bfc3f07367942d8a0fb59c52b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-51.ec2.internal\" (UID: \"c82be4bfc3f07367942d8a0fb59c52b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.998344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.998344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:18.998308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f69fc9e0b3bb68d48b82a7e4d504e25f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal\" (UID: \"f69fc9e0b3bb68d48b82a7e4d504e25f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:18.999231 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:18.999216 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.100223 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.100162 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.158628 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.158607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:19.162639 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.162623 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:19.200853 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.200826 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.301258 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.301225 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.401819 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.401749 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.502182 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.502153 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.535333 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.535308 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 17:52:19.535767 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.535491 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 17:52:19.602793 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.602770 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.615015 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.614984 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:47:19 +0000 UTC" deadline="2028-03-30 04:37:11.255227046 +0000 UTC" Aug 04 17:52:19.615015 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.615009 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14482h44m51.640220587s" Aug 04 17:52:19.628726 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.628711 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:19.703033 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:19.702966 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-51.ec2.internal\" not found" Aug 04 17:52:19.746035 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.746009 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:19.767532 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:19.767491 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82be4bfc3f07367942d8a0fb59c52b4.slice/crio-51546417ee03f14262bd9bb677d8a67ec80458a7682168c8617b1749477c3509 WatchSource:0}: Error finding container 51546417ee03f14262bd9bb677d8a67ec80458a7682168c8617b1749477c3509: Status 404 returned error can't find the container with id 51546417ee03f14262bd9bb677d8a67ec80458a7682168c8617b1749477c3509 Aug 04 17:52:19.767960 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:19.767939 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69fc9e0b3bb68d48b82a7e4d504e25f.slice/crio-a6a2554a4098e0b764e78d6d8cada5b1bdc3375704a64dce68554f24e05a6a91 WatchSource:0}: Error finding container a6a2554a4098e0b764e78d6d8cada5b1bdc3375704a64dce68554f24e05a6a91: Status 404 returned error can't find the container with id a6a2554a4098e0b764e78d6d8cada5b1bdc3375704a64dce68554f24e05a6a91 Aug 04 17:52:19.773423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.773407 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 17:52:19.794785 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.794751 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" Aug 04 17:52:19.809404 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.809381 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 17:52:19.810221 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.810206 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" Aug 04 17:52:19.819509 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.819488 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 17:52:19.899916 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:19.899893 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:20.352493 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.352401 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:20.574606 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.574571 2559 apiserver.go:52] "Watching apiserver" Aug 04 17:52:20.582121 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.582097 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 17:52:20.582504 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.582479 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-c5dg8","openshift-network-operator/iptables-alerter-mfpft","openshift-ovn-kubernetes/ovnkube-node-jzvlk","kube-system/global-pull-secret-syncer-fwwnn","kube-system/konnectivity-agent-xbr79","kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc","openshift-image-registry/node-ca-66lz4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal","openshift-multus/multus-8vbnl","openshift-multus/multus-additional-cni-plugins-p6jwf","openshift-cluster-node-tuning-operator/tuned-f66fp","openshift-dns/node-resolver-mm72d","openshift-multus/network-metrics-daemon-hxvcv"] Aug 04 17:52:20.585253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.585228 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.587584 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.587557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.590305 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.590282 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.590419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.590376 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kb4vj\"" Aug 04 17:52:20.591247 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.591216 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 17:52:20.591850 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.591824 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.591952 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.591853 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.591952 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.591901 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z2l9x\"" Aug 04 17:52:20.591952 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.591923 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.592133 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.592099 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.594498 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.594442 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 17:52:20.594974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.594952 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 17:52:20.596196 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.595724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 17:52:20.596196 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.595734 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.596196 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.595768 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.596196 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.595976 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 17:52:20.596893 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.596875 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.596983 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.596932 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:20.597284 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.597262 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh62v\"" Aug 04 17:52:20.598048 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.598026 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 17:52:20.599571 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.599528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.603014 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.601490 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.603014 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.601579 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:20.603014 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.601965 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 17:52:20.603014 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.602266 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w96m9\"" Aug 04 17:52:20.603014 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.602270 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 17:52:20.604601 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.604583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.607279 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-kubelet\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-systemd-units\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-ovn\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-script-lib\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.607490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607420 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-netns\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607443 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-systemd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607466 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-bin\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgntt\" (UniqueName: \"kubernetes.io/projected/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-kube-api-access-sgntt\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-netd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-hosts-file\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607566 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607624 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-env-overrides\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607675 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-sys-fs\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.607693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckmq\" (UniqueName: \"kubernetes.io/projected/78594f32-accc-496d-b898-534d012aa7d4-kube-api-access-kckmq\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-tmp-dir\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-var-lib-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607785 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-etc-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-node-log\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovn-node-metrics-cert\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-slash\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.607933 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-config\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.607990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ntq\" (UniqueName: \"kubernetes.io/projected/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-kube-api-access-l2ntq\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-dbus\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-socket-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-device-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608062 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cxk77\"" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608108 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608138 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-kubelet-config\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.608198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608166 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-registration-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.608644 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608283 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-log-socket\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.608644 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.608644 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.608358 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.609149 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.609125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:20.609237 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.609189 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:20.610085 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.610048 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 17:52:20.613078 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.613060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.616853 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.616799 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.617234 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.617209 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.617447 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.617429 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-82cwc\"" Aug 04 17:52:20.617703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.617677 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 17:52:20.618871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.618846 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.620992 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.620971 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.621290 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.621275 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 17:52:20.621390 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.621292 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qbnb6\"" Aug 04 17:52:20.621570 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.621512 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.621642 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.621613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.621763 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.621748 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 17:52:20.624227 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.624204 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmq7d\"" Aug 04 17:52:20.624326 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.624231 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 17:52:20.624392 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.624378 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 17:52:20.624999 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.624980 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.627345 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.627324 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 17:52:20.627430 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.627357 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:52:20.627498 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.627357 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nbcb8\"" Aug 04 17:52:20.638888 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.638865 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r29rj" Aug 04 17:52:20.648859 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.648835 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r29rj" Aug 04 17:52:20.696488 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.696458 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 17:52:20.709155 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-slash\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ntq\" (UniqueName: \"kubernetes.io/projected/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-kube-api-access-l2ntq\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-dbus\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22ced487-9535-409a-b8df-db07acb91705-iptables-alerter-script\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.709312 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-hostroot\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-run\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-socket-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-os-release\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-host\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-system-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-tuned\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-registration-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-log-socket\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-agent-certs\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-tmp\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-script-lib\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cni-binary-copy\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.709678 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-bin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-etc-kubernetes\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-netns\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-serviceca\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-modprobe-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-lib-modules\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709823 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-var-lib-kubelet\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgntt\" (UniqueName: \"kubernetes.io/projected/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-kube-api-access-sgntt\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-netd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709908 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9sj9\" (UniqueName: \"kubernetes.io/projected/0b8e2e81-3287-47a4-98ce-df337087e4e6-kube-api-access-t9sj9\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-socket-dir-parent\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.709992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-conf\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-sys\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-host\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710077 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-env-overrides\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.710423 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710153 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-multus-certs\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-sys-fs\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckmq\" (UniqueName: \"kubernetes.io/projected/78594f32-accc-496d-b898-534d012aa7d4-kube-api-access-kckmq\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-tmp-dir\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-node-log\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22ced487-9535-409a-b8df-db07acb91705-host-slash\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-cnibin\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-os-release\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-config\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-kubelet\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-conf-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxxj\" (UniqueName: \"kubernetes.io/projected/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-kube-api-access-6xxxj\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-device-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.711344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-kubelet-config\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-multus\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysconfig\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-system-cni-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-k8s-cni-cncf-io\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-kubelet\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-systemd-units\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-ovn\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dm5\" (UniqueName: \"kubernetes.io/projected/e99de878-d603-4183-8a3c-4bec85213382-kube-api-access-86dm5\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cnibin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.710965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-systemd\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-systemd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-bin\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sjl\" (UniqueName: \"kubernetes.io/projected/22ced487-9535-409a-b8df-db07acb91705-kube-api-access-q7sjl\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b77n\" (UniqueName: \"kubernetes.io/projected/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-kube-api-access-6b77n\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.712150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4flz\" (UniqueName: \"kubernetes.io/projected/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-kube-api-access-t4flz\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-konnectivity-ca\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-hosts-file\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-var-lib-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-etc-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovn-node-metrics-cert\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-netns\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-daemon-config\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-kubernetes\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-slash\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.711962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-dbus\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-socket-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-registration-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-log-socket\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.712984 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-netd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.712790 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712841 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-ovn-kubernetes\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-run-netns\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.712879 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:21.212839073 +0000 UTC m=+2.954993133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-sys-fs\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.712984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-script-lib\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-device-dir\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-cni-bin\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-systemd\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713258 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/78594f32-accc-496d-b898-534d012aa7d4-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713281 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-env-overrides\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-host-kubelet\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713335 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2875059c-8a6a-424b-ba19-66e66511f8de-kubelet-config\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-run-ovn\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714016 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713439 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-tmp-dir\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-etc-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-hosts-file\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-var-lib-openvswitch\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-node-log\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-systemd-units\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovnkube-config\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.714881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.713770 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 17:52:20.719392 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.719370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-ovn-node-metrics-cert\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.728331 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.728306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckmq\" (UniqueName: \"kubernetes.io/projected/78594f32-accc-496d-b898-534d012aa7d4-kube-api-access-kckmq\") pod \"aws-ebs-csi-driver-node-pmqhc\" (UID: \"78594f32-accc-496d-b898-534d012aa7d4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.729403 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.729380 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ntq\" (UniqueName: \"kubernetes.io/projected/30c7c655-cdba-48bb-8bce-2fdf9acc96a8-kube-api-access-l2ntq\") pod \"ovnkube-node-jzvlk\" (UID: \"30c7c655-cdba-48bb-8bce-2fdf9acc96a8\") " pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.730636 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.730611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgntt\" (UniqueName: \"kubernetes.io/projected/ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c-kube-api-access-sgntt\") pod \"node-resolver-mm72d\" (UID: \"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c\") " pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.739281 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.739232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" event={"ID":"f69fc9e0b3bb68d48b82a7e4d504e25f","Type":"ContainerStarted","Data":"a6a2554a4098e0b764e78d6d8cada5b1bdc3375704a64dce68554f24e05a6a91"} Aug 04 17:52:20.740794 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.740767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" event={"ID":"c82be4bfc3f07367942d8a0fb59c52b4","Type":"ContainerStarted","Data":"51546417ee03f14262bd9bb677d8a67ec80458a7682168c8617b1749477c3509"} Aug 04 17:52:20.812522 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:20.812690 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cni-binary-copy\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.812690 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-bin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.812690 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-etc-kubernetes\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.812690 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-serviceca\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812691 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-bin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-etc-kubernetes\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-modprobe-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-lib-modules\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-var-lib-kubelet\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9sj9\" (UniqueName: \"kubernetes.io/projected/0b8e2e81-3287-47a4-98ce-df337087e4e6-kube-api-access-t9sj9\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812826 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-modprobe-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.812871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-socket-dir-parent\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-conf\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-sys\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-lib-modules\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-host\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-var-lib-kubelet\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-multus-certs\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.812968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-socket-dir-parent\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-multus-certs\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813070 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-conf\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813125 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-host\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-sys\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22ced487-9535-409a-b8df-db07acb91705-host-slash\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.813285 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-cnibin\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22ced487-9535-409a-b8df-db07acb91705-host-slash\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813332 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-cnibin\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-os-release\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-kubelet\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-conf-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cni-binary-copy\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813436 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-kubelet\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxxj\" (UniqueName: \"kubernetes.io/projected/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-kube-api-access-6xxxj\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-os-release\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-conf-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-multus\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysconfig\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-system-cni-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-var-lib-cni-multus\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.813920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-k8s-cni-cncf-io\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysconfig\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813606 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dm5\" (UniqueName: \"kubernetes.io/projected/e99de878-d603-4183-8a3c-4bec85213382-kube-api-access-86dm5\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cnibin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-system-cni-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-systemd\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-k8s-cni-cncf-io\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-serviceca\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sjl\" (UniqueName: \"kubernetes.io/projected/22ced487-9535-409a-b8df-db07acb91705-kube-api-access-q7sjl\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-cnibin\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b77n\" (UniqueName: \"kubernetes.io/projected/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-kube-api-access-6b77n\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-systemd\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4flz\" (UniqueName: \"kubernetes.io/projected/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-kube-api-access-t4flz\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-konnectivity-ca\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.814811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-netns\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-daemon-config\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813975 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-kubernetes\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-host-run-netns\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22ced487-9535-409a-b8df-db07acb91705-iptables-alerter-script\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-kubernetes\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.813926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-sysctl-d\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-hostroot\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-run\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-os-release\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-host\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-run\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-system-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-tuned\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-hostroot\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815361 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-agent-certs\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-tmp\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814319 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-system-cni-dir\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e99de878-d603-4183-8a3c-4bec85213382-os-release\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-host\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.814391 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e99de878-d603-4183-8a3c-4bec85213382-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-konnectivity-ca\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.814444 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:21.314426419 +0000 UTC m=+3.056580476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814511 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-multus-daemon-config\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.815987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.814668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22ced487-9535-409a-b8df-db07acb91705-iptables-alerter-script\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.816939 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.816912 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-etc-tuned\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.817189 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.817170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac47e548-6eab-4a87-af8f-89dbe85ee4f4-agent-certs\") pod \"konnectivity-agent-xbr79\" (UID: \"ac47e548-6eab-4a87-af8f-89dbe85ee4f4\") " pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.817546 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.817522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-tmp\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.820782 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.820759 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:20.820894 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.820793 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:20.820894 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.820808 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:20.820894 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:20.820886 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:21.320861588 +0000 UTC m=+3.063015653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:20.824134 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.824101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9sj9\" (UniqueName: \"kubernetes.io/projected/0b8e2e81-3287-47a4-98ce-df337087e4e6-kube-api-access-t9sj9\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:20.824636 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.824616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxxj\" (UniqueName: \"kubernetes.io/projected/fe50a6e9-b81d-4e3e-81e9-942ecde2bbca-kube-api-access-6xxxj\") pod \"multus-8vbnl\" (UID: \"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca\") " pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.827862 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.827836 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sjl\" (UniqueName: \"kubernetes.io/projected/22ced487-9535-409a-b8df-db07acb91705-kube-api-access-q7sjl\") pod \"iptables-alerter-mfpft\" (UID: \"22ced487-9535-409a-b8df-db07acb91705\") " pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.828075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.828022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4flz\" (UniqueName: \"kubernetes.io/projected/b81b0fd6-2589-4cd7-9eab-8a12fb60dff6-kube-api-access-t4flz\") pod \"tuned-f66fp\" (UID: \"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6\") " pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.830073 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.830050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dm5\" (UniqueName: \"kubernetes.io/projected/e99de878-d603-4183-8a3c-4bec85213382-kube-api-access-86dm5\") pod \"multus-additional-cni-plugins-p6jwf\" (UID: \"e99de878-d603-4183-8a3c-4bec85213382\") " pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.832809 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.832785 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b77n\" (UniqueName: \"kubernetes.io/projected/e6805167-3bd3-4ebd-8f5f-7af6d2537f56-kube-api-access-6b77n\") pod \"node-ca-66lz4\" (UID: \"e6805167-3bd3-4ebd-8f5f-7af6d2537f56\") " pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.904091 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.903983 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" Aug 04 17:52:20.916296 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.916268 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78594f32_accc_496d_b898_534d012aa7d4.slice/crio-7997f312c426c3952ea42a96f35fd05943248c7889d27bf6bde3d71b6347d007 WatchSource:0}: Error finding container 7997f312c426c3952ea42a96f35fd05943248c7889d27bf6bde3d71b6347d007: Status 404 returned error can't find the container with id 7997f312c426c3952ea42a96f35fd05943248c7889d27bf6bde3d71b6347d007 Aug 04 17:52:20.920639 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.920617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mm72d" Aug 04 17:52:20.928848 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.928819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:20.930669 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.930643 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2c6d80_d62d_4ec6_9e08_1524b1d75a5c.slice/crio-5c39c87d149e0006e4993e992269bb556ced452c98230f03b5f063d021701ba5 WatchSource:0}: Error finding container 5c39c87d149e0006e4993e992269bb556ced452c98230f03b5f063d021701ba5: Status 404 returned error can't find the container with id 5c39c87d149e0006e4993e992269bb556ced452c98230f03b5f063d021701ba5 Aug 04 17:52:20.935398 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.935376 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:20.940429 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.940396 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c7c655_cdba_48bb_8bce_2fdf9acc96a8.slice/crio-93bd9f6f3a7d1a5a4e8861f636d112643ff7b697fa62f3e97bdc65797d42652c WatchSource:0}: Error finding container 93bd9f6f3a7d1a5a4e8861f636d112643ff7b697fa62f3e97bdc65797d42652c: Status 404 returned error can't find the container with id 93bd9f6f3a7d1a5a4e8861f636d112643ff7b697fa62f3e97bdc65797d42652c Aug 04 17:52:20.941703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.941620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-66lz4" Aug 04 17:52:20.946463 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.946438 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac47e548_6eab_4a87_af8f_89dbe85ee4f4.slice/crio-fb380d004e2afc48e4a60386ee8c544156386401d6f9aa5fea2a1abdc335c43e WatchSource:0}: Error finding container fb380d004e2afc48e4a60386ee8c544156386401d6f9aa5fea2a1abdc335c43e: Status 404 returned error can't find the container with id fb380d004e2afc48e4a60386ee8c544156386401d6f9aa5fea2a1abdc335c43e Aug 04 17:52:20.950488 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.950460 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mfpft" Aug 04 17:52:20.954749 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.954395 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6805167_3bd3_4ebd_8f5f_7af6d2537f56.slice/crio-135afe6f89efba9a960edd14b3d601ffe72e26a70621570567488433ed5238fd WatchSource:0}: Error finding container 135afe6f89efba9a960edd14b3d601ffe72e26a70621570567488433ed5238fd: Status 404 returned error can't find the container with id 135afe6f89efba9a960edd14b3d601ffe72e26a70621570567488433ed5238fd Aug 04 17:52:20.960603 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.960566 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8vbnl" Aug 04 17:52:20.961385 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.961357 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ced487_9535_409a_b8df_db07acb91705.slice/crio-a772299663412999d32931fae2f58f14cdd75fb97aa133bea6bcf3cf8bbc5fc1 WatchSource:0}: Error finding container a772299663412999d32931fae2f58f14cdd75fb97aa133bea6bcf3cf8bbc5fc1: Status 404 returned error can't find the container with id a772299663412999d32931fae2f58f14cdd75fb97aa133bea6bcf3cf8bbc5fc1 Aug 04 17:52:20.967654 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.967634 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" Aug 04 17:52:20.971620 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.971457 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe50a6e9_b81d_4e3e_81e9_942ecde2bbca.slice/crio-c73c8c687f8c1e5b855200d18baaade2cdf5c9213cd712af82afaa9a72202d43 WatchSource:0}: Error finding container c73c8c687f8c1e5b855200d18baaade2cdf5c9213cd712af82afaa9a72202d43: Status 404 returned error can't find the container with id c73c8c687f8c1e5b855200d18baaade2cdf5c9213cd712af82afaa9a72202d43 Aug 04 17:52:20.974367 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:20.974345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f66fp" Aug 04 17:52:20.977492 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.977467 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99de878_d603_4183_8a3c_4bec85213382.slice/crio-cb8b179f4ea7fd0c7286449f8de166777b59056408430c30cde5b786c846632d WatchSource:0}: Error finding container cb8b179f4ea7fd0c7286449f8de166777b59056408430c30cde5b786c846632d: Status 404 returned error can't find the container with id cb8b179f4ea7fd0c7286449f8de166777b59056408430c30cde5b786c846632d Aug 04 17:52:20.984341 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:52:20.984317 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81b0fd6_2589_4cd7_9eab_8a12fb60dff6.slice/crio-84001fe988c8fbcd3e467b880f4c14bcb7be94836ee7bed9fc9be2e852fbabd9 WatchSource:0}: Error finding container 84001fe988c8fbcd3e467b880f4c14bcb7be94836ee7bed9fc9be2e852fbabd9: Status 404 returned error can't find the container with id 84001fe988c8fbcd3e467b880f4c14bcb7be94836ee7bed9fc9be2e852fbabd9 Aug 04 17:52:21.217480 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.217379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:21.217679 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.217502 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:21.217679 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.217599 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:22.217575458 +0000 UTC m=+3.959729520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:21.318075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.318034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:21.318258 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.318209 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:21.318312 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.318291 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:22.318266644 +0000 UTC m=+4.060420682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:21.419362 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.419330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:21.419576 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.419554 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:21.419673 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.419587 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:21.419673 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.419603 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:21.419673 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:21.419669 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:22.419648786 +0000 UTC m=+4.161802840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:21.652909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.652750 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:47:21 +0000 UTC" deadline="2028-03-04 08:22:46.674603053 +0000 UTC" Aug 04 17:52:21.652909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.652812 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13862h30m25.021796063s" Aug 04 17:52:21.745395 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.745292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerStarted","Data":"cb8b179f4ea7fd0c7286449f8de166777b59056408430c30cde5b786c846632d"} Aug 04 17:52:21.750645 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.750453 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vbnl" event={"ID":"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca","Type":"ContainerStarted","Data":"c73c8c687f8c1e5b855200d18baaade2cdf5c9213cd712af82afaa9a72202d43"} Aug 04 17:52:21.752863 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.752833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbr79" event={"ID":"ac47e548-6eab-4a87-af8f-89dbe85ee4f4","Type":"ContainerStarted","Data":"fb380d004e2afc48e4a60386ee8c544156386401d6f9aa5fea2a1abdc335c43e"} Aug 04 17:52:21.755034 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.755008 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" event={"ID":"78594f32-accc-496d-b898-534d012aa7d4","Type":"ContainerStarted","Data":"7997f312c426c3952ea42a96f35fd05943248c7889d27bf6bde3d71b6347d007"} Aug 04 17:52:21.757177 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.757148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f66fp" event={"ID":"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6","Type":"ContainerStarted","Data":"84001fe988c8fbcd3e467b880f4c14bcb7be94836ee7bed9fc9be2e852fbabd9"} Aug 04 17:52:21.759198 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.759175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mfpft" event={"ID":"22ced487-9535-409a-b8df-db07acb91705","Type":"ContainerStarted","Data":"a772299663412999d32931fae2f58f14cdd75fb97aa133bea6bcf3cf8bbc5fc1"} Aug 04 17:52:21.761872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.761849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66lz4" event={"ID":"e6805167-3bd3-4ebd-8f5f-7af6d2537f56","Type":"ContainerStarted","Data":"135afe6f89efba9a960edd14b3d601ffe72e26a70621570567488433ed5238fd"} Aug 04 17:52:21.764049 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.764025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"93bd9f6f3a7d1a5a4e8861f636d112643ff7b697fa62f3e97bdc65797d42652c"} Aug 04 17:52:21.765478 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:21.765455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mm72d" event={"ID":"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c","Type":"ContainerStarted","Data":"5c39c87d149e0006e4993e992269bb556ced452c98230f03b5f063d021701ba5"} Aug 04 17:52:22.227713 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.227681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:22.227860 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.227831 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:22.227927 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.227901 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:24.227879102 +0000 UTC m=+5.970033159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:22.328646 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.328605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:22.328860 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.328766 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:22.328936 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.328915 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:24.328891375 +0000 UTC m=+6.071045433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:22.429318 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.429280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:22.429528 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.429498 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:22.429630 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.429543 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:22.429630 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.429559 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:22.429630 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.429624 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:24.429605062 +0000 UTC m=+6.171759119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:22.653841 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.653766 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:47:21 +0000 UTC" deadline="2028-05-12 14:44:08.98453055 +0000 UTC" Aug 04 17:52:22.653841 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.653808 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15524h51m46.330727499s" Aug 04 17:52:22.735799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.735766 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:22.735965 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.735913 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:22.736327 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.736307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:22.736422 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.736403 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:22.736519 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:22.736465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:22.736597 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:22.736532 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:23.779145 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:23.779049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" event={"ID":"f69fc9e0b3bb68d48b82a7e4d504e25f","Type":"ContainerStarted","Data":"e380e9515875a084968eb1672426d6674175451ba73cdab9a5898021a80578cd"} Aug 04 17:52:23.783037 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:23.782999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" event={"ID":"c82be4bfc3f07367942d8a0fb59c52b4","Type":"ContainerStarted","Data":"c23e0064e983bafd13e39f47c0b43c74969788ca3863a479deea8360eb777b3e"} Aug 04 17:52:24.245062 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.245026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:24.245252 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.245219 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:24.245312 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.245281 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.245261351 +0000 UTC m=+9.987415403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:24.346148 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.346109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:24.346353 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.346326 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:24.346419 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.346398 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.346376396 +0000 UTC m=+10.088530451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:24.447310 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.447272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:24.447604 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.447482 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:24.447604 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.447511 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:24.447604 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.447524 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:24.447604 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.447608 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.447587622 +0000 UTC m=+10.189741678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:24.736083 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.736055 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:24.736275 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.736179 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:24.736620 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.736555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:24.736744 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.736669 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:24.736744 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:24.736733 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:24.736856 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:24.736807 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:25.791247 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:25.791208 2559 generic.go:356] "Generic (PLEG): container finished" podID="f69fc9e0b3bb68d48b82a7e4d504e25f" containerID="e380e9515875a084968eb1672426d6674175451ba73cdab9a5898021a80578cd" exitCode=0 Aug 04 17:52:25.791730 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:25.791269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" event={"ID":"f69fc9e0b3bb68d48b82a7e4d504e25f","Type":"ContainerDied","Data":"e380e9515875a084968eb1672426d6674175451ba73cdab9a5898021a80578cd"} Aug 04 17:52:25.815248 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:25.815169 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" podStartSLOduration=5.815151344 podStartE2EDuration="5.815151344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:23.816055645 +0000 UTC m=+5.558209706" watchObservedRunningTime="2026-08-04 17:52:25.815151344 +0000 UTC m=+7.557305399" Aug 04 17:52:26.735877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:26.735834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:26.735877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:26.735857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:26.736222 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:26.736005 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:26.736222 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:26.736019 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:26.736222 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:26.736205 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:26.736406 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:26.736360 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:28.283121 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.283079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:28.283652 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.283282 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:28.283652 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.283349 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:36.283329334 +0000 UTC m=+18.025483388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:28.383832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.383800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:28.384005 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.383953 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:28.384056 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.384023 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:36.384004559 +0000 UTC m=+18.126158597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:28.484664 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.484629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:28.484862 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.484832 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:28.484862 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.484859 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:28.485008 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.484877 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:28.485008 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.484941 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:36.484922281 +0000 UTC m=+18.227076326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:28.734477 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.734439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:28.734477 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.734465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:28.734723 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.734604 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:28.734723 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.734697 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:28.734850 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:28.734757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:28.734850 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:28.734830 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:30.734636 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:30.734605 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:30.734636 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:30.734632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:30.735080 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:30.734731 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:30.735080 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:30.735069 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:30.735157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:30.735115 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:30.735199 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:30.735162 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:32.733868 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:32.733837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:32.733868 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:32.733871 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:32.734330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:32.733842 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:32.734330 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:32.733952 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:32.734330 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:32.734053 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:32.734330 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:32.734105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:34.733354 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:34.733313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:34.733354 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:34.733328 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:34.733772 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:34.733421 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:34.733772 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:34.733465 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:34.733772 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:34.733538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:34.733772 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:34.733634 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:36.352869 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.352775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:36.353338 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.352919 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:36.353338 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.352983 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:52:52.352963863 +0000 UTC m=+34.095117902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:36.453377 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.453346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:36.453550 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.453518 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:36.453618 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.453591 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:52.4535693 +0000 UTC m=+34.195723351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:36.553908 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.553874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:36.554069 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.554014 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:36.554069 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.554038 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:36.554069 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.554052 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:36.554189 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.554110 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:52.554096406 +0000 UTC m=+34.296250443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:36.732941 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.732860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:36.733106 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.732868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:36.733106 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:36.732999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:36.733106 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.732992 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:36.740172 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.733124 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:36.740172 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:36.733208 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:38.734991 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:38.734954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:38.735487 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:38.735410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:38.737610 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:38.735803 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:38.737610 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:38.736446 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:38.737610 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:38.736537 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:38.737610 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:38.736692 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:40.733640 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.733371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:40.734259 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.733402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:40.734259 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.733402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:40.734259 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:40.734003 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:40.734259 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:40.734171 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:40.734477 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:40.734322 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:40.820695 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.820670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" event={"ID":"f69fc9e0b3bb68d48b82a7e4d504e25f","Type":"ContainerStarted","Data":"cca3c7e18af25422102b9341349ffbe6a0980aa21eab7d15e85fb75442d55df8"} Aug 04 17:52:40.822053 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.822028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f66fp" event={"ID":"b81b0fd6-2589-4cd7-9eab-8a12fb60dff6","Type":"ContainerStarted","Data":"edff7ef8bc92bc04c1cfaa81dce8e514b757d59be8f4a4b2204747670281bf76"} Aug 04 17:52:40.823642 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.823605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-66lz4" event={"ID":"e6805167-3bd3-4ebd-8f5f-7af6d2537f56","Type":"ContainerStarted","Data":"360986e11cfde2744d1a554a9ec49b7b4835e1bb70c10e7b0c7a79ab315dfd7c"} Aug 04 17:52:40.825733 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.825711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"4a8868fe899e4aba95ad73fdab4c26ef58fc6fda298ca3f4188f9dc80ffc87fb"} Aug 04 17:52:40.825815 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.825760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"412b24e0e1c24d90cc7726a69f990eaae51880d1bd731b8e259b5347fbe95c2b"} Aug 04 17:52:40.825815 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.825788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"d1b61f5d42eb6eec6db81aa89e354084737e05993bb5dfd859499e0602eb3311"} Aug 04 17:52:40.826977 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.826958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mm72d" event={"ID":"ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c","Type":"ContainerStarted","Data":"7c030c343eef15f666e2a3e086938e8761ac6af6d83c0c60384ee494ee1c5625"} Aug 04 17:52:40.828352 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.828330 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="a2682c40ca47672520ecbe925de9f3e4d4805da211f28f4a19c25873127526a1" exitCode=0 Aug 04 17:52:40.828439 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.828399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"a2682c40ca47672520ecbe925de9f3e4d4805da211f28f4a19c25873127526a1"} Aug 04 17:52:40.829752 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.829683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8vbnl" event={"ID":"fe50a6e9-b81d-4e3e-81e9-942ecde2bbca","Type":"ContainerStarted","Data":"0734c6afa45e66c9db3bfabdc7e17147c6cde51a3cc2db8f6adef68a0ab72cf9"} Aug 04 17:52:40.831135 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.831102 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbr79" event={"ID":"ac47e548-6eab-4a87-af8f-89dbe85ee4f4","Type":"ContainerStarted","Data":"8f21d79d30256f54b66122d79aa5a33c7cab618c4f7112aafd3259440b1c5f6a"} Aug 04 17:52:40.832732 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.832657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" event={"ID":"78594f32-accc-496d-b898-534d012aa7d4","Type":"ContainerStarted","Data":"f120ca2f0fe5fdc504a784e7fe39c347de7882e16ca4454fa613245a37c41864"} Aug 04 17:52:40.838490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.838444 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-51.ec2.internal" podStartSLOduration=20.838428137 podStartE2EDuration="20.838428137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.838033057 +0000 UTC m=+22.580187115" watchObservedRunningTime="2026-08-04 17:52:40.838428137 +0000 UTC m=+22.580582196" Aug 04 17:52:40.857895 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.857858 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8vbnl" podStartSLOduration=2.516805506 podStartE2EDuration="21.857845829s" totalImagesPullingTime="19.341040323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.857380966 +0000 UTC m=+22.599535028" watchObservedRunningTime="2026-08-04 17:52:40.857845829 +0000 UTC m=+22.599999887" Aug 04 17:52:40.901244 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.901194 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f66fp" podStartSLOduration=2.61535597 podStartE2EDuration="21.901174608s" totalImagesPullingTime="19.285818638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.900572443 +0000 UTC m=+22.642726504" watchObservedRunningTime="2026-08-04 17:52:40.901174608 +0000 UTC m=+22.643328669" Aug 04 17:52:40.923522 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.923456 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xbr79" podStartSLOduration=2.603720419 podStartE2EDuration="21.923440957s" totalImagesPullingTime="19.319720538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.922034448 +0000 UTC m=+22.664188506" watchObservedRunningTime="2026-08-04 17:52:40.923440957 +0000 UTC m=+22.665595015" Aug 04 17:52:40.950991 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.950924 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mm72d" podStartSLOduration=2.635816294 podStartE2EDuration="21.950909276s" totalImagesPullingTime="19.315092982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.950728691 +0000 UTC m=+22.692882748" watchObservedRunningTime="2026-08-04 17:52:40.950909276 +0000 UTC m=+22.693063334" Aug 04 17:52:40.979376 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:40.979331 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-66lz4" podStartSLOduration=2.66626398 podStartE2EDuration="21.979319016s" totalImagesPullingTime="19.313055036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:40.978628886 +0000 UTC m=+22.720782941" watchObservedRunningTime="2026-08-04 17:52:40.979319016 +0000 UTC m=+22.721473102" Aug 04 17:52:41.722953 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.722922 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 17:52:41.839055 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.838771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"3ace157c9f162a93fc164228650eebf1b22d8139c0cdb334f6e825b252b529de"} Aug 04 17:52:41.839055 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.838921 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"4211ef1d05848b718df48b7be06582b0d4c427f5860852d445f58097880e5671"} Aug 04 17:52:41.839055 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.838968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"5dd56f0af20eca3a62f25c864acab2e671262ce742b2321a268e4d1e4b95a3a3"} Aug 04 17:52:41.840709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.840676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" event={"ID":"78594f32-accc-496d-b898-534d012aa7d4","Type":"ContainerStarted","Data":"bcc905bac418adfec4f3c55ee933872ec6d56e79f8f74f3565239569437d196f"} Aug 04 17:52:41.842137 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.842071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mfpft" event={"ID":"22ced487-9535-409a-b8df-db07acb91705","Type":"ContainerStarted","Data":"c647950bc4021703f1e1b16d510d7597039f238300e38c47b844b211010acdb0"} Aug 04 17:52:41.866659 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:41.866614 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mfpft" podStartSLOduration=11.66854723 podStartE2EDuration="22.866601227s" totalImagesPullingTime="11.198053997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:41.866282427 +0000 UTC m=+23.608436484" watchObservedRunningTime="2026-08-04 17:52:41.866601227 +0000 UTC m=+23.608755284" Aug 04 17:52:42.663321 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.663219 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T17:52:41.722948924Z","UUID":"98f007d8-18b8-41a4-bd1d-64d4b7fc1c76","Handler":null,"Name":"","Endpoint":""} Aug 04 17:52:42.665311 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.665283 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 17:52:42.665439 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.665320 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 17:52:42.680231 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.680206 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:42.680754 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.680737 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:42.736737 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.736710 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:42.736901 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.736815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:42.736901 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:42.736828 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:42.736901 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:42.736847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:42.737059 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:42.736921 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:42.737059 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:42.736991 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:43.854695 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:43.854646 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" event={"ID":"78594f32-accc-496d-b898-534d012aa7d4","Type":"ContainerStarted","Data":"99d1d5f726b4a9633e3aaad4c24cfdc3c25ddda6f6d21d3b2ee43d0c57b20998"} Aug 04 17:52:43.858560 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:43.858537 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:43.858936 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:43.858901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"9560bcb2b293d2f45948730f622d8f474e298a7150a4aeecc0a748c34fe36d5b"} Aug 04 17:52:44.686951 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:44.686758 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mm72d_ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c/dns-node-resolver/0.log" Aug 04 17:52:44.736931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:44.736896 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:44.737084 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:44.736939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:44.737084 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:44.736939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:44.737084 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:44.737018 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:44.738089 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:44.737104 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:44.738089 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:44.737170 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:45.248301 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.248277 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66lz4_e6805167-3bd3-4ebd-8f5f-7af6d2537f56/node-ca/0.log" Aug 04 17:52:45.865900 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.865851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" event={"ID":"30c7c655-cdba-48bb-8bce-2fdf9acc96a8","Type":"ContainerStarted","Data":"636dde830186128bd5c8625601dd0938a4157904dd8b98cf488d53300ccb3063"} Aug 04 17:52:45.866141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.866115 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:45.867695 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.867666 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="018a3213d7699a3e7d5fa9b670c579e9bab7c36d84f378fd4104cd4b197f291b" exitCode=0 Aug 04 17:52:45.867809 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.867711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"018a3213d7699a3e7d5fa9b670c579e9bab7c36d84f378fd4104cd4b197f291b"} Aug 04 17:52:45.882369 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.882351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:45.895310 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.895262 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmqhc" podStartSLOduration=9.509675084 podStartE2EDuration="26.895248364s" totalImagesPullingTime="17.38557328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:43.876573638 +0000 UTC m=+25.618727695" watchObservedRunningTime="2026-08-04 17:52:45.895248364 +0000 UTC m=+27.637402421" Aug 04 17:52:45.895782 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:45.895755 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" podStartSLOduration=7.354781184 podStartE2EDuration="26.895745943s" totalImagesPullingTime="19.540964759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:45.895126732 +0000 UTC m=+27.637280789" watchObservedRunningTime="2026-08-04 17:52:45.895745943 +0000 UTC m=+27.637900019" Aug 04 17:52:46.733156 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.732938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:46.733858 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.732938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:46.733858 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:46.733260 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:46.733858 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:46.733443 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:46.733858 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.732978 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:46.733858 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:46.733569 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:46.869836 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.869726 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:46.870177 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.870155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:46.885314 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:46.885294 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:47.874443 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.874408 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="e05c0b0f27bd1701135eca249d9d383e78d0e9a33f1f5ac1f7be87ab6c18ad4b" exitCode=0 Aug 04 17:52:47.874910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.874517 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"e05c0b0f27bd1701135eca249d9d383e78d0e9a33f1f5ac1f7be87ab6c18ad4b"} Aug 04 17:52:47.874910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.874585 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:47.877471 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.877450 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fwwnn"] Aug 04 17:52:47.877588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.877566 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:47.877689 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:47.877663 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:47.879724 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.879698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c5dg8"] Aug 04 17:52:47.879841 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.879798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:47.879909 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:47.879891 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:47.880365 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.880346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hxvcv"] Aug 04 17:52:47.880456 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:47.880445 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:47.881398 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:47.880599 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:48.835082 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:48.835048 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:48.835260 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:48.835166 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:48.835670 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:48.835649 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xbr79" Aug 04 17:52:48.877056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:48.877025 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:49.733405 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:49.733235 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:49.733573 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:49.733318 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:49.733573 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:49.733328 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:49.733573 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:49.733558 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:49.733696 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:49.733592 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:49.733696 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:49.733490 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:49.882402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:49.882366 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="2ba52f8216851f63eb80b47cdd396d4b876ad8af05213a4269cf45b011a9b6cf" exitCode=0 Aug 04 17:52:49.882775 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:49.882443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"2ba52f8216851f63eb80b47cdd396d4b876ad8af05213a4269cf45b011a9b6cf"} Aug 04 17:52:50.490268 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:50.490234 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:52:50.490493 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:50.490480 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Aug 04 17:52:50.505163 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:50.505122 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" podUID="30c7c655-cdba-48bb-8bce-2fdf9acc96a8" containerName="ovnkube-controller" probeResult="failure" output="" Aug 04 17:52:50.515193 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:50.515158 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" podUID="30c7c655-cdba-48bb-8bce-2fdf9acc96a8" containerName="ovnkube-controller" probeResult="failure" output="" Aug 04 17:52:51.732842 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:51.732812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:51.732842 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:51.732846 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:51.733239 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:51.732911 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:51.733239 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:51.732925 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:51.733239 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:51.732983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:51.733239 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:51.733042 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:52.375175 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:52.375122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:52.375391 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.375291 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:52.375391 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.375381 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret podName:2875059c-8a6a-424b-ba19-66e66511f8de nodeName:}" failed. No retries permitted until 2026-08-04 17:53:24.375347766 +0000 UTC m=+66.117501802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret") pod "global-pull-secret-syncer-fwwnn" (UID: "2875059c-8a6a-424b-ba19-66e66511f8de") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:52.476455 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:52.476419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:52.476658 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.476598 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:52.476729 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.476669 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs podName:0b8e2e81-3287-47a4-98ce-df337087e4e6 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:24.476649974 +0000 UTC m=+66.218804033 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs") pod "network-metrics-daemon-hxvcv" (UID: "0b8e2e81-3287-47a4-98ce-df337087e4e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:52.577737 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:52.577707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:52.577903 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.577843 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:52.577903 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.577855 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:52.577903 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.577864 2559 projected.go:196] Error preparing data for projected volume kube-api-access-b297z for pod openshift-network-diagnostics/network-check-target-c5dg8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:52.578012 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:52.577907 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z podName:a07152cc-aee6-41e4-9403-3d7a354e5570 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:24.577895737 +0000 UTC m=+66.320049773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b297z" (UniqueName: "kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z") pod "network-check-target-c5dg8" (UID: "a07152cc-aee6-41e4-9403-3d7a354e5570") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:53.733530 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:53.733483 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:53.734008 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:53.733492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:53.734008 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:53.733621 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:53.734008 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:53.733639 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:53.734008 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:53.733714 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:53.734008 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:53.733797 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:54.896994 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:54.896959 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="0c53334564cf29d28ad5b5ddb82abea43b0c860138b21ef5f32d845717eb3c2c" exitCode=0 Aug 04 17:52:54.897355 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:54.897004 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"0c53334564cf29d28ad5b5ddb82abea43b0c860138b21ef5f32d845717eb3c2c"} Aug 04 17:52:55.733710 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:55.733670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:55.733710 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:55.733705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:55.733901 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:55.733671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:55.733901 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:55.733783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:55.733901 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:55.733859 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:55.733991 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:55.733938 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:55.901625 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:55.901596 2559 generic.go:356] "Generic (PLEG): container finished" podID="e99de878-d603-4183-8a3c-4bec85213382" containerID="8c41ab65b71d56ff7eb7c1617e3f99ffa1f9da6555a4516979845868c785a258" exitCode=0 Aug 04 17:52:55.902136 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:55.901633 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerDied","Data":"8c41ab65b71d56ff7eb7c1617e3f99ffa1f9da6555a4516979845868c785a258"} Aug 04 17:52:56.906323 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:56.906292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" event={"ID":"e99de878-d603-4183-8a3c-4bec85213382","Type":"ContainerStarted","Data":"22bd6ea2aebefa4d9912efe4b142cdb27021fad31e716eb341cbef47105111e0"} Aug 04 17:52:56.964303 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:56.964254 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p6jwf" podStartSLOduration=7.220541881 podStartE2EDuration="37.964238679s" totalImagesPullingTime="30.743696798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:56.963771394 +0000 UTC m=+38.705925451" watchObservedRunningTime="2026-08-04 17:52:56.964238679 +0000 UTC m=+38.706392758" Aug 04 17:52:57.733048 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:57.732972 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:57.733048 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:57.733018 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:57.733048 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:57.733022 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:57.733237 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:57.733095 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:52:57.733237 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:57.733164 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:57.733301 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:57.733247 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:59.733770 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:59.733740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:52:59.734152 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:59.733740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:52:59.734152 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:59.733836 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:52:59.734152 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:52:59.733740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:52:59.734152 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:59.733910 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:52:59.734152 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:52:59.733969 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:53:01.733491 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:01.733461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:01.733874 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:01.733571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:01.733874 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:01.733582 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:53:01.733874 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:01.733651 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:53:01.733874 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:01.733660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:01.733874 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:01.733721 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:53:03.733393 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:03.733352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:03.733879 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:03.733466 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:53:03.733879 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:03.733352 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:03.733879 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:03.733366 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:03.733879 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:03.733582 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:53:03.733879 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:03.733675 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:53:05.732794 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:05.732761 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:05.732794 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:05.732779 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:05.733201 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:05.732863 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:53:05.733201 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:05.732883 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:05.733201 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:05.732965 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:53:05.733201 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:05.733018 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:53:07.732940 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:07.732906 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:07.733461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:07.733016 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:07.733461 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:07.733030 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fwwnn" podUID="2875059c-8a6a-424b-ba19-66e66511f8de" Aug 04 17:53:07.733461 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:07.733132 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c5dg8" podUID="a07152cc-aee6-41e4-9403-3d7a354e5570" Aug 04 17:53:07.733461 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:07.733167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:07.733461 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:07.733260 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxvcv" podUID="0b8e2e81-3287-47a4-98ce-df337087e4e6" Aug 04 17:53:09.136910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.136839 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-51.ec2.internal" event="NodeReady" Aug 04 17:53:09.137247 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.136963 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 17:53:09.177171 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.177141 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c97cb6f48-s6xxc"] Aug 04 17:53:09.196083 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.196057 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c97cb6f48-s6xxc"] Aug 04 17:53:09.196241 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.196161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.198715 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.198691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 17:53:09.198858 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.198734 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 17:53:09.199008 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.198995 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kwcpz\"" Aug 04 17:53:09.199058 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.199034 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 17:53:09.205264 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.205241 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8ttb2"] Aug 04 17:53:09.206978 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.206959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 17:53:09.220117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.220096 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-722jp"] Aug 04 17:53:09.220255 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.220238 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.227039 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.227021 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z8v6x\"" Aug 04 17:53:09.227111 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.227025 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 17:53:09.227205 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.227190 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 17:53:09.227424 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.227405 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 17:53:09.234154 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.234132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-722jp"] Aug 04 17:53:09.234236 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.234164 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ttb2"] Aug 04 17:53:09.234236 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.234184 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d4klp"] Aug 04 17:53:09.234236 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.234230 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.236894 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.236877 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4v5p\"" Aug 04 17:53:09.236986 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.236926 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 17:53:09.237397 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.237380 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 17:53:09.256210 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.256187 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d4klp"] Aug 04 17:53:09.256298 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.256286 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.258590 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.258568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwp6w\"" Aug 04 17:53:09.258728 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.258644 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 17:53:09.258871 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.258854 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 17:53:09.258932 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.258856 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 17:53:09.258932 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.258861 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 17:53:09.297358 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297329 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-image-registry-private-configuration\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.297358 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297356 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b955947d-6c9c-423d-9ac3-f6eeb9449497-config-volume\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b955947d-6c9c-423d-9ac3-f6eeb9449497-metrics-tls\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297399 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-registry-tls\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-cert\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297457 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-installation-pull-secrets\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmgx\" (UniqueName: \"kubernetes.io/projected/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-kube-api-access-plmgx\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.297581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b955947d-6c9c-423d-9ac3-f6eeb9449497-tmp-dir\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm26\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-kube-api-access-rwm26\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgfh\" (UniqueName: \"kubernetes.io/projected/b955947d-6c9c-423d-9ac3-f6eeb9449497-kube-api-access-4mgfh\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/911a9056-9587-41aa-b68e-2a2bd3a7a21e-crio-socket\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/911a9056-9587-41aa-b68e-2a2bd3a7a21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-bound-sa-token\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297755 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4zb\" (UniqueName: \"kubernetes.io/projected/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-api-access-4f4zb\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.297860 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-registry-certificates\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.298130 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/911a9056-9587-41aa-b68e-2a2bd3a7a21e-data-volume\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.298130 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3895845c-8fc8-4876-9f26-001964bfdca1-ca-trust-extracted\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.298130 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.297974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-trusted-ca\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398722 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-bound-sa-token\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398722 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4zb\" (UniqueName: \"kubernetes.io/projected/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-api-access-4f4zb\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.398722 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-registry-certificates\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/911a9056-9587-41aa-b68e-2a2bd3a7a21e-data-volume\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3895845c-8fc8-4876-9f26-001964bfdca1-ca-trust-extracted\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-trusted-ca\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-image-registry-private-configuration\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b955947d-6c9c-423d-9ac3-f6eeb9449497-config-volume\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b955947d-6c9c-423d-9ac3-f6eeb9449497-metrics-tls\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-registry-tls\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-cert\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.398974 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-installation-pull-secrets\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.398979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmgx\" (UniqueName: \"kubernetes.io/projected/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-kube-api-access-plmgx\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399003 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b955947d-6c9c-423d-9ac3-f6eeb9449497-tmp-dir\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm26\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-kube-api-access-rwm26\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgfh\" (UniqueName: \"kubernetes.io/projected/b955947d-6c9c-423d-9ac3-f6eeb9449497-kube-api-access-4mgfh\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/911a9056-9587-41aa-b68e-2a2bd3a7a21e-crio-socket\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/911a9056-9587-41aa-b68e-2a2bd3a7a21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/911a9056-9587-41aa-b68e-2a2bd3a7a21e-data-volume\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.399450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3895845c-8fc8-4876-9f26-001964bfdca1-ca-trust-extracted\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.399909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b955947d-6c9c-423d-9ac3-f6eeb9449497-config-volume\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.399909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b955947d-6c9c-423d-9ac3-f6eeb9449497-tmp-dir\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.399909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.399790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-registry-certificates\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.400357 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.400331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/911a9056-9587-41aa-b68e-2a2bd3a7a21e-crio-socket\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.400725 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.400705 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3895845c-8fc8-4876-9f26-001964bfdca1-trusted-ca\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.403402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403381 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/911a9056-9587-41aa-b68e-2a2bd3a7a21e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.403402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403397 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-registry-tls\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.403591 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b955947d-6c9c-423d-9ac3-f6eeb9449497-metrics-tls\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.403591 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-image-registry-private-configuration\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.403591 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403574 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-cert\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.403693 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.403623 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3895845c-8fc8-4876-9f26-001964bfdca1-installation-pull-secrets\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.407687 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.407663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4zb\" (UniqueName: \"kubernetes.io/projected/911a9056-9587-41aa-b68e-2a2bd3a7a21e-kube-api-access-4f4zb\") pod \"insights-runtime-extractor-d4klp\" (UID: \"911a9056-9587-41aa-b68e-2a2bd3a7a21e\") " pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.407843 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.407824 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-bound-sa-token\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.408327 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.408293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmgx\" (UniqueName: \"kubernetes.io/projected/5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94-kube-api-access-plmgx\") pod \"ingress-canary-8ttb2\" (UID: \"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94\") " pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.408638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.408622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm26\" (UniqueName: \"kubernetes.io/projected/3895845c-8fc8-4876-9f26-001964bfdca1-kube-api-access-rwm26\") pod \"image-registry-c97cb6f48-s6xxc\" (UID: \"3895845c-8fc8-4876-9f26-001964bfdca1\") " pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.408796 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.408781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgfh\" (UniqueName: \"kubernetes.io/projected/b955947d-6c9c-423d-9ac3-f6eeb9449497-kube-api-access-4mgfh\") pod \"dns-default-722jp\" (UID: \"b955947d-6c9c-423d-9ac3-f6eeb9449497\") " pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.505988 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.505953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:09.530094 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.530069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ttb2" Aug 04 17:53:09.542026 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.541980 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-722jp" Aug 04 17:53:09.564242 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.564216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d4klp" Aug 04 17:53:09.732996 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.732968 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:09.733165 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.733062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:09.733311 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.733244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:09.736566 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.736478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ttb2"] Aug 04 17:53:09.736939 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.736917 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 17:53:09.737061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.736943 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 17:53:09.737061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.737017 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 17:53:09.737061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.737036 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smv6k\"" Aug 04 17:53:09.737672 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.737654 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7jt5\"" Aug 04 17:53:09.737728 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.737659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 17:53:09.740216 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:09.740184 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efcd9cf_4967_4fba_8dd4_6cc4b2d83a94.slice/crio-bc956b73313130d196cc1571e07b93569c3085bd2a018652db2e305c63af7874 WatchSource:0}: Error finding container bc956b73313130d196cc1571e07b93569c3085bd2a018652db2e305c63af7874: Status 404 returned error can't find the container with id bc956b73313130d196cc1571e07b93569c3085bd2a018652db2e305c63af7874 Aug 04 17:53:09.935545 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.935458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ttb2" event={"ID":"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94","Type":"ContainerStarted","Data":"bc956b73313130d196cc1571e07b93569c3085bd2a018652db2e305c63af7874"} Aug 04 17:53:09.953872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.953841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d4klp"] Aug 04 17:53:09.958150 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.958098 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-722jp"] Aug 04 17:53:09.958705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:09.958604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c97cb6f48-s6xxc"] Aug 04 17:53:09.967154 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:09.967125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911a9056_9587_41aa_b68e_2a2bd3a7a21e.slice/crio-2e2c078892d41471a31e6f8acc318c525b08fe235615ad71b7fa4607316432ac WatchSource:0}: Error finding container 2e2c078892d41471a31e6f8acc318c525b08fe235615ad71b7fa4607316432ac: Status 404 returned error can't find the container with id 2e2c078892d41471a31e6f8acc318c525b08fe235615ad71b7fa4607316432ac Aug 04 17:53:09.967898 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:09.967875 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb955947d_6c9c_423d_9ac3_f6eeb9449497.slice/crio-c046e628ddc08a7fcdefadeccd295ed9bba98aae6961162ca5e00ba8d6eefac6 WatchSource:0}: Error finding container c046e628ddc08a7fcdefadeccd295ed9bba98aae6961162ca5e00ba8d6eefac6: Status 404 returned error can't find the container with id c046e628ddc08a7fcdefadeccd295ed9bba98aae6961162ca5e00ba8d6eefac6 Aug 04 17:53:09.968643 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:09.968614 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3895845c_8fc8_4876_9f26_001964bfdca1.slice/crio-4e8d20f7044d426499fc61c0ff4455d08f8b037b1d1a02c2c00f10ed9aea8dfc WatchSource:0}: Error finding container 4e8d20f7044d426499fc61c0ff4455d08f8b037b1d1a02c2c00f10ed9aea8dfc: Status 404 returned error can't find the container with id 4e8d20f7044d426499fc61c0ff4455d08f8b037b1d1a02c2c00f10ed9aea8dfc Aug 04 17:53:10.609308 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.609270 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp"] Aug 04 17:53:10.641338 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.640609 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp"] Aug 04 17:53:10.641338 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.640713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:10.644315 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.643994 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 17:53:10.644315 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.643998 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6wq7l\"" Aug 04 17:53:10.709867 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.709833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-mkmzp\" (UID: \"30a155f1-d79e-43cc-8f08-7b1655785630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:10.810946 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.810909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-mkmzp\" (UID: \"30a155f1-d79e-43cc-8f08-7b1655785630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:10.811106 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:10.811061 2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Aug 04 17:53:10.811159 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:10.811140 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates podName:30a155f1-d79e-43cc-8f08-7b1655785630 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:11.31111823 +0000 UTC m=+53.053272283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-mkmzp" (UID: "30a155f1-d79e-43cc-8f08-7b1655785630") : secret "prometheus-operator-admission-webhook-tls" not found Aug 04 17:53:10.940120 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.940032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d4klp" event={"ID":"911a9056-9587-41aa-b68e-2a2bd3a7a21e","Type":"ContainerStarted","Data":"07f8ad73d465450d925a45cde5cd6b0e6088d5ba66dee40cc9ce7366b2e745cc"} Aug 04 17:53:10.940120 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.940100 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d4klp" event={"ID":"911a9056-9587-41aa-b68e-2a2bd3a7a21e","Type":"ContainerStarted","Data":"2e2c078892d41471a31e6f8acc318c525b08fe235615ad71b7fa4607316432ac"} Aug 04 17:53:10.942392 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.942361 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" event={"ID":"3895845c-8fc8-4876-9f26-001964bfdca1","Type":"ContainerStarted","Data":"961822b90cec6ca330cb3d5dd4703dc140c84edc53a48af397e50e01b41dc9de"} Aug 04 17:53:10.942565 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.942415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" event={"ID":"3895845c-8fc8-4876-9f26-001964bfdca1","Type":"ContainerStarted","Data":"4e8d20f7044d426499fc61c0ff4455d08f8b037b1d1a02c2c00f10ed9aea8dfc"} Aug 04 17:53:10.942565 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.942461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:10.944265 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.944239 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-722jp" event={"ID":"b955947d-6c9c-423d-9ac3-f6eeb9449497","Type":"ContainerStarted","Data":"c046e628ddc08a7fcdefadeccd295ed9bba98aae6961162ca5e00ba8d6eefac6"} Aug 04 17:53:10.994626 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:10.994177 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" podStartSLOduration=3.994156208 podStartE2EDuration="3.994156208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:10.992476003 +0000 UTC m=+52.734630062" watchObservedRunningTime="2026-08-04 17:53:10.994156208 +0000 UTC m=+52.736310267" Aug 04 17:53:11.316255 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:11.316215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-mkmzp\" (UID: \"30a155f1-d79e-43cc-8f08-7b1655785630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:11.318730 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:11.318703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/30a155f1-d79e-43cc-8f08-7b1655785630-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-mkmzp\" (UID: \"30a155f1-d79e-43cc-8f08-7b1655785630\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:11.552288 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:11.552248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:13.119409 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.119357 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp"] Aug 04 17:53:13.124472 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:13.124439 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a155f1_d79e_43cc_8f08_7b1655785630.slice/crio-a0212b87b8921017e6090d54d384d4474895d77e6c47ffcd49dca13ca27787ae WatchSource:0}: Error finding container a0212b87b8921017e6090d54d384d4474895d77e6c47ffcd49dca13ca27787ae: Status 404 returned error can't find the container with id a0212b87b8921017e6090d54d384d4474895d77e6c47ffcd49dca13ca27787ae Aug 04 17:53:13.956846 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.956764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d4klp" event={"ID":"911a9056-9587-41aa-b68e-2a2bd3a7a21e","Type":"ContainerStarted","Data":"6cbd3add9d2f2ac608967b602fa61821e7e58b177dcc923170b5a8aa9e9cc8c9"} Aug 04 17:53:13.958709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.958662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" event={"ID":"30a155f1-d79e-43cc-8f08-7b1655785630","Type":"ContainerStarted","Data":"a0212b87b8921017e6090d54d384d4474895d77e6c47ffcd49dca13ca27787ae"} Aug 04 17:53:13.962600 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.962575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ttb2" event={"ID":"5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94","Type":"ContainerStarted","Data":"c6ab0ca45e845b531d875b4ff22a44224e4568fc081122c60f999b30a9ca8b7d"} Aug 04 17:53:13.965342 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.965319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-722jp" event={"ID":"b955947d-6c9c-423d-9ac3-f6eeb9449497","Type":"ContainerStarted","Data":"e5017150807e41418a31d0194ef3839c7d83a0c7305bf5d272219c0cf2de73ea"} Aug 04 17:53:13.965455 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.965365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-722jp" event={"ID":"b955947d-6c9c-423d-9ac3-f6eeb9449497","Type":"ContainerStarted","Data":"6538e2617a1a3438972485b10415adfe50e9b1a6799767008548a7f37e5905fd"} Aug 04 17:53:13.965605 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.965579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-722jp" Aug 04 17:53:13.980900 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:13.980835 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8ttb2" podStartSLOduration=1.742459091 podStartE2EDuration="4.980818881s" totalImagesPullingTime="3.23835979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:13.980475243 +0000 UTC m=+55.722629301" watchObservedRunningTime="2026-08-04 17:53:13.980818881 +0000 UTC m=+55.722972939" Aug 04 17:53:14.001774 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:14.001724 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-722jp" podStartSLOduration=2.000242993 podStartE2EDuration="5.001704436s" totalImagesPullingTime="3.001461443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:14.00104409 +0000 UTC m=+55.743198149" watchObservedRunningTime="2026-08-04 17:53:14.001704436 +0000 UTC m=+55.743858496" Aug 04 17:53:14.970794 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:14.970696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d4klp" event={"ID":"911a9056-9587-41aa-b68e-2a2bd3a7a21e","Type":"ContainerStarted","Data":"cad922bd21c35d1bea05922621a3b632953c9241b3abc0fc361f4b39aedf41da"} Aug 04 17:53:14.972233 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:14.972211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" event={"ID":"30a155f1-d79e-43cc-8f08-7b1655785630","Type":"ContainerStarted","Data":"95819d0f68d6e6eec457af2a1d48f323a256d4ea8ee1d553c0814dd54eaf3ae7"} Aug 04 17:53:14.972380 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:14.972365 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:14.976867 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:14.976848 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" Aug 04 17:53:15.012521 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:15.012460 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d4klp" podStartSLOduration=1.526611548 podStartE2EDuration="6.012445907s" totalImagesPullingTime="4.485834359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:53:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:14.993483632 +0000 UTC m=+56.735637689" watchObservedRunningTime="2026-08-04 17:53:15.012445907 +0000 UTC m=+56.754599965" Aug 04 17:53:15.012620 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:15.012580 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-mkmzp" podStartSLOduration=3.430616659 podStartE2EDuration="5.012573182s" totalImagesPullingTime="1.581956523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:15.012200345 +0000 UTC m=+56.754354403" watchObservedRunningTime="2026-08-04 17:53:15.012573182 +0000 UTC m=+56.754727239" Aug 04 17:53:20.515246 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:20.515219 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jzvlk" Aug 04 17:53:22.198709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.198667 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5"] Aug 04 17:53:22.201589 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.201571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.204627 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.204606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 17:53:22.205193 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.205175 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 17:53:22.205256 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.205212 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 17:53:22.205318 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.205218 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 17:53:22.205536 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.205489 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 17:53:22.205646 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.205579 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4kc5s\"" Aug 04 17:53:22.209027 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.209011 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh"] Aug 04 17:53:22.210799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.210787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.213456 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.213437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Aug 04 17:53:22.214295 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.214273 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p7nqf\"" Aug 04 17:53:22.214450 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.214430 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Aug 04 17:53:22.214601 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.214582 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Aug 04 17:53:22.215251 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.215236 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jbsxj"] Aug 04 17:53:22.217473 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.217457 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5"] Aug 04 17:53:22.217584 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.217573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.219908 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.219892 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7kr7n\"" Aug 04 17:53:22.220991 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.220976 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 17:53:22.221079 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.220976 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 17:53:22.225644 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.225628 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 17:53:22.226100 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.226080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh"] Aug 04 17:53:22.302347 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/faf2b139-3acf-4b43-a573-df0e930845fa-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.302347 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-wtmp\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-metrics-client-ca\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-sys\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-textfile\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302750 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwt6w\" (UniqueName: \"kubernetes.io/projected/12b99295-bad7-40f7-b3ae-359ae84592a4-kube-api-access-jwt6w\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-root\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302800 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kj9\" (UniqueName: \"kubernetes.io/projected/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-kube-api-access-x9kj9\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.302910 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.302845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlg8\" (UniqueName: \"kubernetes.io/projected/faf2b139-3acf-4b43-a573-df0e930845fa-kube-api-access-nqlg8\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.403518 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-textfile\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.403518 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwt6w\" (UniqueName: \"kubernetes.io/projected/12b99295-bad7-40f7-b3ae-359ae84592a4-kube-api-access-jwt6w\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-root\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kj9\" (UniqueName: \"kubernetes.io/projected/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-kube-api-access-x9kj9\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlg8\" (UniqueName: \"kubernetes.io/projected/faf2b139-3acf-4b43-a573-df0e930845fa-kube-api-access-nqlg8\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/faf2b139-3acf-4b43-a573-df0e930845fa-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-root\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.403747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-wtmp\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-textfile\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:22.403872 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-metrics-client-ca\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-wtmp\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.403919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:22.403940 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls podName:12b99295-bad7-40f7-b3ae-359ae84592a4 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:22.903919118 +0000 UTC m=+64.646073155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls") pod "node-exporter-jbsxj" (UID: "12b99295-bad7-40f7-b3ae-359ae84592a4") : secret "node-exporter-tls" not found Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/faf2b139-3acf-4b43-a573-df0e930845fa-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.404056 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:22.404038 2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:53:22.404103 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls podName:faf2b139-3acf-4b43-a573-df0e930845fa nodeName:}" failed. No retries permitted until 2026-08-04 17:53:22.90408949 +0000 UTC m=+64.646243526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-pqvwh" (UID: "faf2b139-3acf-4b43-a573-df0e930845fa") : secret "kube-state-metrics-tls" not found Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-sys\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.404579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12b99295-bad7-40f7-b3ae-359ae84592a4-sys\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.405020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-metrics-client-ca\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.405020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.405020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.404838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.405177 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.405109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.405329 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.405248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faf2b139-3acf-4b43-a573-df0e930845fa-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.406893 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.406872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.406893 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.406885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.407017 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.406926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.407082 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.407062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.414905 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.414888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlg8\" (UniqueName: \"kubernetes.io/projected/faf2b139-3acf-4b43-a573-df0e930845fa-kube-api-access-nqlg8\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.415158 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.415142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kj9\" (UniqueName: \"kubernetes.io/projected/5a223ae7-ebbe-4ab4-a002-7a3606c5e301-kube-api-access-x9kj9\") pod \"openshift-state-metrics-6bb4d7b4dd-52pm5\" (UID: \"5a223ae7-ebbe-4ab4-a002-7a3606c5e301\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.417036 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.417015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwt6w\" (UniqueName: \"kubernetes.io/projected/12b99295-bad7-40f7-b3ae-359ae84592a4-kube-api-access-jwt6w\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.510861 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.510771 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" Aug 04 17:53:22.645485 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.645463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5"] Aug 04 17:53:22.647312 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:22.647285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a223ae7_ebbe_4ab4_a002_7a3606c5e301.slice/crio-acaf2d333722ed23ed7c92fe2309a63d99bf93a01e32405ad64ce081e7d83d25 WatchSource:0}: Error finding container acaf2d333722ed23ed7c92fe2309a63d99bf93a01e32405ad64ce081e7d83d25: Status 404 returned error can't find the container with id acaf2d333722ed23ed7c92fe2309a63d99bf93a01e32405ad64ce081e7d83d25 Aug 04 17:53:22.908311 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.908280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.908490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.908321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.910728 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.910699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12b99295-bad7-40f7-b3ae-359ae84592a4-node-exporter-tls\") pod \"node-exporter-jbsxj\" (UID: \"12b99295-bad7-40f7-b3ae-359ae84592a4\") " pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:22.910827 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.910815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/faf2b139-3acf-4b43-a573-df0e930845fa-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-pqvwh\" (UID: \"faf2b139-3acf-4b43-a573-df0e930845fa\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:22.999526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.997180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" event={"ID":"5a223ae7-ebbe-4ab4-a002-7a3606c5e301","Type":"ContainerStarted","Data":"1b020e4353f9824f7f137749727f942acfddb037c341e91c4231efca956c23f2"} Aug 04 17:53:22.999526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.997295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" event={"ID":"5a223ae7-ebbe-4ab4-a002-7a3606c5e301","Type":"ContainerStarted","Data":"3b0d03a958d7c95e010f9c6e6349d5dd03793280449b3f332c76a3bf2b80799e"} Aug 04 17:53:22.999526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:22.997336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" event={"ID":"5a223ae7-ebbe-4ab4-a002-7a3606c5e301","Type":"ContainerStarted","Data":"acaf2d333722ed23ed7c92fe2309a63d99bf93a01e32405ad64ce081e7d83d25"} Aug 04 17:53:23.119694 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:23.119659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" Aug 04 17:53:23.125609 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:23.125587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jbsxj" Aug 04 17:53:23.133896 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:23.133867 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b99295_bad7_40f7_b3ae_359ae84592a4.slice/crio-471f5d84161d13ea4d3e998789fb22adb562493e5688977f6f18086712914edf WatchSource:0}: Error finding container 471f5d84161d13ea4d3e998789fb22adb562493e5688977f6f18086712914edf: Status 404 returned error can't find the container with id 471f5d84161d13ea4d3e998789fb22adb562493e5688977f6f18086712914edf Aug 04 17:53:23.247204 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:23.247178 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh"] Aug 04 17:53:23.249918 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:23.249895 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf2b139_3acf_4b43_a573_df0e930845fa.slice/crio-2a65a1930493461813c846c90fd5b8834af4b344b3c390f8c537187b2f0db6b9 WatchSource:0}: Error finding container 2a65a1930493461813c846c90fd5b8834af4b344b3c390f8c537187b2f0db6b9: Status 404 returned error can't find the container with id 2a65a1930493461813c846c90fd5b8834af4b344b3c390f8c537187b2f0db6b9 Aug 04 17:53:23.975303 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:23.975274 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-722jp" Aug 04 17:53:24.003265 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.003230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" event={"ID":"faf2b139-3acf-4b43-a573-df0e930845fa","Type":"ContainerStarted","Data":"2a65a1930493461813c846c90fd5b8834af4b344b3c390f8c537187b2f0db6b9"} Aug 04 17:53:24.005076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.005041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbsxj" event={"ID":"12b99295-bad7-40f7-b3ae-359ae84592a4","Type":"ContainerStarted","Data":"471f5d84161d13ea4d3e998789fb22adb562493e5688977f6f18086712914edf"} Aug 04 17:53:24.381102 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.381066 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:53:24.384223 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.384187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.388384 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.388365 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 17:53:24.388491 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.388365 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 17:53:24.388684 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.388655 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 17:53:24.388801 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.388722 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 17:53:24.389359 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.389339 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4kwvm\"" Aug 04 17:53:24.389445 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.389345 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 17:53:24.389534 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.389485 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 17:53:24.389613 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.389594 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 17:53:24.408923 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.408895 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:53:24.422082 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.422063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:24.424861 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.424842 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 17:53:24.435166 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.435146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2875059c-8a6a-424b-ba19-66e66511f8de-original-pull-secret\") pod \"global-pull-secret-syncer-fwwnn\" (UID: \"2875059c-8a6a-424b-ba19-66e66511f8de\") " pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:24.451265 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.451248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fwwnn" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgmh\" (UniqueName: \"kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.523117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.522961 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.526308 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.526239 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 17:53:24.536211 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.535928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b8e2e81-3287-47a4-98ce-df337087e4e6-metrics-certs\") pod \"network-metrics-daemon-hxvcv\" (UID: \"0b8e2e81-3287-47a4-98ce-df337087e4e6\") " pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:24.613342 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.612793 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fwwnn"] Aug 04 17:53:24.619555 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:24.619529 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2875059c_8a6a_424b_ba19_66e66511f8de.slice/crio-2c1c4074a11908f1d88c15fb90f82df103ae22aef476bf1df1e864f213501a8a WatchSource:0}: Error finding container 2c1c4074a11908f1d88c15fb90f82df103ae22aef476bf1df1e864f213501a8a: Status 404 returned error can't find the container with id 2c1c4074a11908f1d88c15fb90f82df103ae22aef476bf1df1e864f213501a8a Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgmh\" (UniqueName: \"kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.623604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.624792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.624732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.625756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.625283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.625867 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.625846 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.628962 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.626567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.629925 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.629634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.629925 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.629732 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 17:53:24.638929 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.638861 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 17:53:24.639613 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.639570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgmh\" (UniqueName: \"kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh\") pod \"console-d76487d64-rxhn7\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.649760 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.649690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b297z\" (UniqueName: \"kubernetes.io/projected/a07152cc-aee6-41e4-9403-3d7a354e5570-kube-api-access-b297z\") pod \"network-check-target-c5dg8\" (UID: \"a07152cc-aee6-41e4-9403-3d7a354e5570\") " pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:24.694978 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.694951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:24.749276 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.748835 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7jt5\"" Aug 04 17:53:24.756440 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.755517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:24.758975 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.758937 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smv6k\"" Aug 04 17:53:24.768606 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.768588 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxvcv" Aug 04 17:53:24.849078 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.849046 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:53:24.853088 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:24.853054 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf4036c_1f50_4772_b22a_152d2df0b640.slice/crio-913fbf51d5951286326c2ebd04695c50fafa65d4faae4d8f8789494f2b4195d0 WatchSource:0}: Error finding container 913fbf51d5951286326c2ebd04695c50fafa65d4faae4d8f8789494f2b4195d0: Status 404 returned error can't find the container with id 913fbf51d5951286326c2ebd04695c50fafa65d4faae4d8f8789494f2b4195d0 Aug 04 17:53:24.915666 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.915646 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c5dg8"] Aug 04 17:53:24.917921 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:24.917892 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07152cc_aee6_41e4_9403_3d7a354e5570.slice/crio-6bbe9af9ca90b57ce8ab310602466e94da38c673b78149c5d82f943e7a7f1a8d WatchSource:0}: Error finding container 6bbe9af9ca90b57ce8ab310602466e94da38c673b78149c5d82f943e7a7f1a8d: Status 404 returned error can't find the container with id 6bbe9af9ca90b57ce8ab310602466e94da38c673b78149c5d82f943e7a7f1a8d Aug 04 17:53:24.932187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:24.932162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hxvcv"] Aug 04 17:53:24.935790 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:24.935770 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8e2e81_3287_47a4_98ce_df337087e4e6.slice/crio-c8d4955ce635b1f01afb2795decd690ff4ed0ff2bfa875ce546c1a98f5e02adb WatchSource:0}: Error finding container c8d4955ce635b1f01afb2795decd690ff4ed0ff2bfa875ce546c1a98f5e02adb: Status 404 returned error can't find the container with id c8d4955ce635b1f01afb2795decd690ff4ed0ff2bfa875ce546c1a98f5e02adb Aug 04 17:53:25.011138 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.011100 2559 generic.go:356] "Generic (PLEG): container finished" podID="12b99295-bad7-40f7-b3ae-359ae84592a4" containerID="cf0bc6463b9dc5eab7561f2303b614df5100c35c10d8b21c534cfa38de51f203" exitCode=0 Aug 04 17:53:25.011308 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.011226 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbsxj" event={"ID":"12b99295-bad7-40f7-b3ae-359ae84592a4","Type":"ContainerDied","Data":"cf0bc6463b9dc5eab7561f2303b614df5100c35c10d8b21c534cfa38de51f203"} Aug 04 17:53:25.019011 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.018979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" event={"ID":"5a223ae7-ebbe-4ab4-a002-7a3606c5e301","Type":"ContainerStarted","Data":"787148100aee13b7231095321ef091b9403fab8d9ee7f80d2f97c7f7789ddd60"} Aug 04 17:53:25.021111 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.021089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" event={"ID":"faf2b139-3acf-4b43-a573-df0e930845fa","Type":"ContainerStarted","Data":"46094baaff8803f5fa00b5b4c4000e2e8693b6fc3ffc0daf38d3ef0598192b05"} Aug 04 17:53:25.021210 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.021132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" event={"ID":"faf2b139-3acf-4b43-a573-df0e930845fa","Type":"ContainerStarted","Data":"d3a44b8bd5af9267e10f4a966e25ed13cf17653688dfb8c583f8fd496977801a"} Aug 04 17:53:25.021210 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.021150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" event={"ID":"faf2b139-3acf-4b43-a573-df0e930845fa","Type":"ContainerStarted","Data":"6f75960df71bf8f2c3e79ad6e8d02e8a954bbaa2f7ada9779c48d2417cfbe570"} Aug 04 17:53:25.022317 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.022293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c5dg8" event={"ID":"a07152cc-aee6-41e4-9403-3d7a354e5570","Type":"ContainerStarted","Data":"6bbe9af9ca90b57ce8ab310602466e94da38c673b78149c5d82f943e7a7f1a8d"} Aug 04 17:53:25.023409 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.023386 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxvcv" event={"ID":"0b8e2e81-3287-47a4-98ce-df337087e4e6","Type":"ContainerStarted","Data":"c8d4955ce635b1f01afb2795decd690ff4ed0ff2bfa875ce546c1a98f5e02adb"} Aug 04 17:53:25.024670 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.024638 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d76487d64-rxhn7" event={"ID":"caf4036c-1f50-4772-b22a-152d2df0b640","Type":"ContainerStarted","Data":"913fbf51d5951286326c2ebd04695c50fafa65d4faae4d8f8789494f2b4195d0"} Aug 04 17:53:25.025821 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.025753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fwwnn" event={"ID":"2875059c-8a6a-424b-ba19-66e66511f8de","Type":"ContainerStarted","Data":"2c1c4074a11908f1d88c15fb90f82df103ae22aef476bf1df1e864f213501a8a"} Aug 04 17:53:25.072735 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.072695 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-52pm5" podStartSLOduration=1.366956314 podStartE2EDuration="3.072680491s" totalImagesPullingTime="1.705724177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:25.072613925 +0000 UTC m=+66.814767982" watchObservedRunningTime="2026-08-04 17:53:25.072680491 +0000 UTC m=+66.814834548" Aug 04 17:53:25.102842 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:25.102801 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-pqvwh" podStartSLOduration=1.89535673 podStartE2EDuration="3.102783115s" totalImagesPullingTime="1.207426385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:25.102390178 +0000 UTC m=+66.844544253" watchObservedRunningTime="2026-08-04 17:53:25.102783115 +0000 UTC m=+66.844937173" Aug 04 17:53:26.035308 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.035134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbsxj" event={"ID":"12b99295-bad7-40f7-b3ae-359ae84592a4","Type":"ContainerStarted","Data":"ba59e182ae9ec7149b4ba8c9626322d09c59a238298f2e4aeb6df59bc12ad299"} Aug 04 17:53:26.035308 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.035202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jbsxj" event={"ID":"12b99295-bad7-40f7-b3ae-359ae84592a4","Type":"ContainerStarted","Data":"6428111485f10b4cec85962e0cfa90234309f36b6bf06b4eec764e7ca1f03e33"} Aug 04 17:53:26.064679 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.064615 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jbsxj" podStartSLOduration=2.654539316 podStartE2EDuration="4.064594111s" totalImagesPullingTime="1.322209695s" totalInitContainerRuntime="87.8451ms" isStatefulPod=true podCreationTimestamp="2026-08-04 17:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:26.064037122 +0000 UTC m=+67.806191179" watchObservedRunningTime="2026-08-04 17:53:26.064594111 +0000 UTC m=+67.806748170" Aug 04 17:53:26.673298 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.672144 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6dff57d5f6-7q2nx"] Aug 04 17:53:26.691681 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.691651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dff57d5f6-7q2nx"] Aug 04 17:53:26.691830 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.691775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.699267 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.698788 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 17:53:26.699267 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.699057 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bjmwx\"" Aug 04 17:53:26.699267 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.699087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 17:53:26.699523 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.699350 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 17:53:26.699586 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.699569 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5posa4km4aar7\"" Aug 04 17:53:26.699890 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.699712 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 17:53:26.845294 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845257 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-client-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-metrics-server-audit-profiles\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-tls\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-client-certs\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnc2\" (UniqueName: \"kubernetes.io/projected/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-kube-api-access-7vnc2\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.845449 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.845437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-audit-log\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946707 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946707 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-audit-log\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946912 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-client-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946912 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946748 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-metrics-server-audit-profiles\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946912 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-tls\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946912 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-client-certs\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.946912 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.946833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnc2\" (UniqueName: \"kubernetes.io/projected/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-kube-api-access-7vnc2\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.947703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.947679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.947937 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.947921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-audit-log\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.949379 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.949335 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-metrics-server-audit-profiles\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.951486 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.951464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-client-certs\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.952153 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.952093 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-client-ca-bundle\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.952153 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.952110 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-secret-metrics-server-tls\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:26.959323 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:26.959280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnc2\" (UniqueName: \"kubernetes.io/projected/14dbb70e-ee82-484e-b7fa-5d47b6f57ca6-kube-api-access-7vnc2\") pod \"metrics-server-6dff57d5f6-7q2nx\" (UID: \"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6\") " pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:27.009563 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:27.009347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:27.043287 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:27.043205 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxvcv" event={"ID":"0b8e2e81-3287-47a4-98ce-df337087e4e6","Type":"ContainerStarted","Data":"0c93c458a2f6b7e583182be250ad7c490c1467c9766714b7b082b487070793de"} Aug 04 17:53:30.701547 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:30.701515 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6dff57d5f6-7q2nx"] Aug 04 17:53:30.705391 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:30.705353 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dbb70e_ee82_484e_b7fa_5d47b6f57ca6.slice/crio-df0ea876d36d8d457d7ca39c9d78f237bfc01a13affcece82cd0cc7ae6606b19 WatchSource:0}: Error finding container df0ea876d36d8d457d7ca39c9d78f237bfc01a13affcece82cd0cc7ae6606b19: Status 404 returned error can't find the container with id df0ea876d36d8d457d7ca39c9d78f237bfc01a13affcece82cd0cc7ae6606b19 Aug 04 17:53:31.059703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.059666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c5dg8" event={"ID":"a07152cc-aee6-41e4-9403-3d7a354e5570","Type":"ContainerStarted","Data":"72abf170645774b3da7e52af67bd3087d0edff9045fe1fcbf249af6721aef8f4"} Aug 04 17:53:31.059899 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.059786 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:53:31.061413 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.061388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxvcv" event={"ID":"0b8e2e81-3287-47a4-98ce-df337087e4e6","Type":"ContainerStarted","Data":"71cdcc318481708241cc3a73ad56691686588b1082930ff264861a3dc371bb87"} Aug 04 17:53:31.066409 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.066371 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d76487d64-rxhn7" event={"ID":"caf4036c-1f50-4772-b22a-152d2df0b640","Type":"ContainerStarted","Data":"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c"} Aug 04 17:53:31.067895 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.067873 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fwwnn" event={"ID":"2875059c-8a6a-424b-ba19-66e66511f8de","Type":"ContainerStarted","Data":"434471b3ae0242744afe2190b83820f83a27b78bc3a064efcc64ef70713ae636"} Aug 04 17:53:31.068953 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.068938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" event={"ID":"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6","Type":"ContainerStarted","Data":"df0ea876d36d8d457d7ca39c9d78f237bfc01a13affcece82cd0cc7ae6606b19"} Aug 04 17:53:31.088687 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.088639 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c5dg8" podStartSLOduration=66.465606798 podStartE2EDuration="1m12.088626639s" totalImagesPullingTime="5.623019841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:31.088051367 +0000 UTC m=+72.830205427" watchObservedRunningTime="2026-08-04 17:53:31.088626639 +0000 UTC m=+72.830780675" Aug 04 17:53:31.144289 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.144233 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d76487d64-rxhn7" podStartSLOduration=1.457912598 podStartE2EDuration="7.144218557s" totalImagesPullingTime="5.686305959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:31.14376875 +0000 UTC m=+72.885922807" watchObservedRunningTime="2026-08-04 17:53:31.144218557 +0000 UTC m=+72.886372614" Aug 04 17:53:31.213386 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.213328 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hxvcv" podStartSLOduration=70.409366968 podStartE2EDuration="1m12.213310587s" totalImagesPullingTime="1.803943619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:31.211599297 +0000 UTC m=+72.953753355" watchObservedRunningTime="2026-08-04 17:53:31.213310587 +0000 UTC m=+72.955464645" Aug 04 17:53:31.213573 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.213445 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fwwnn" podStartSLOduration=66.276128243 podStartE2EDuration="1m12.213438841s" totalImagesPullingTime="5.937310598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:31.185804367 +0000 UTC m=+72.927958424" watchObservedRunningTime="2026-08-04 17:53:31.213438841 +0000 UTC m=+72.955592898" Aug 04 17:53:31.952054 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:31.952026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c97cb6f48-s6xxc" Aug 04 17:53:33.077779 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.077745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" event={"ID":"14dbb70e-ee82-484e-b7fa-5d47b6f57ca6","Type":"ContainerStarted","Data":"0f2b52c48caff3bb5af10060a518fbdd2963de0764000373f67f27abaedd6547"} Aug 04 17:53:33.102700 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.102660 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" podStartSLOduration=5.621363186 podStartE2EDuration="7.102643544s" totalImagesPullingTime="1.481280358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:33.102139907 +0000 UTC m=+74.844293965" watchObservedRunningTime="2026-08-04 17:53:33.102643544 +0000 UTC m=+74.844797602" Aug 04 17:53:33.889242 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.889208 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:53:33.892429 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.892413 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:33.900975 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.900943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 17:53:33.904305 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:33.904279 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:53:34.011237 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011237 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqh45\" (UniqueName: \"kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011273 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011400 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.011469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.011425 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.112491 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.112491 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112544 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113021 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.112753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqh45\" (UniqueName: \"kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113293 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.113271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113364 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.113271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113364 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.113328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.113637 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.113611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.115674 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.115647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.115752 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.115647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.122544 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.122524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqh45\" (UniqueName: \"kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45\") pod \"console-7ff678c6d8-stn9p\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.204220 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.204145 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:34.348705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.348684 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:53:34.350904 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:53:34.350876 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode388a65e_d08b_4c8b_b17f_04f24f674473.slice/crio-5355cbf92e8848a575d035a5e2676cab84eabfc7076e0c4d007ffb4671439643 WatchSource:0}: Error finding container 5355cbf92e8848a575d035a5e2676cab84eabfc7076e0c4d007ffb4671439643: Status 404 returned error can't find the container with id 5355cbf92e8848a575d035a5e2676cab84eabfc7076e0c4d007ffb4671439643 Aug 04 17:53:34.695758 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.695717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:34.695935 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.695780 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:34.700238 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:34.700218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:35.087077 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:35.087042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff678c6d8-stn9p" event={"ID":"e388a65e-d08b-4c8b-b17f-04f24f674473","Type":"ContainerStarted","Data":"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5"} Aug 04 17:53:35.087255 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:35.087104 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff678c6d8-stn9p" event={"ID":"e388a65e-d08b-4c8b-b17f-04f24f674473","Type":"ContainerStarted","Data":"5355cbf92e8848a575d035a5e2676cab84eabfc7076e0c4d007ffb4671439643"} Aug 04 17:53:35.090658 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:35.090637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:53:35.211994 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:35.211937 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ff678c6d8-stn9p" podStartSLOduration=2.211920863 podStartE2EDuration="2.211920863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:35.160307779 +0000 UTC m=+76.902461837" watchObservedRunningTime="2026-08-04 17:53:35.211920863 +0000 UTC m=+76.954074920" Aug 04 17:53:44.204959 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:44.204846 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:44.204959 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:44.204915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:44.209691 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:44.209670 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:45.125944 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:45.125916 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:53:45.199950 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:45.199920 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:53:47.009972 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:47.009932 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:53:47.010370 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:53:47.009999 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:54:02.076168 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:02.076137 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c5dg8" Aug 04 17:54:07.015526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:07.015475 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:54:07.019220 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:07.019186 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6dff57d5f6-7q2nx" Aug 04 17:54:10.219055 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.218990 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d76487d64-rxhn7" podUID="caf4036c-1f50-4772-b22a-152d2df0b640" containerName="console" containerID="cri-o://983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c" gracePeriod=15 Aug 04 17:54:10.453063 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.453041 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:54:10.480377 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480303 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480377 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480352 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480611 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480611 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480494 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480611 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480589 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480773 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480645 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfgmh\" (UniqueName: \"kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh\") pod \"caf4036c-1f50-4772-b22a-152d2df0b640\" (UID: \"caf4036c-1f50-4772-b22a-152d2df0b640\") " Aug 04 17:54:10.480911 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480802 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:10.480911 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480892 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config" (OuterVolumeSpecName: "console-config") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:10.481026 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.480982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca" (OuterVolumeSpecName: "service-ca") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:10.482857 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.482830 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:10.482961 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.482855 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh" (OuterVolumeSpecName: "kube-api-access-gfgmh") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "kube-api-access-gfgmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:54:10.483000 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.482976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "caf4036c-1f50-4772-b22a-152d2df0b640" (UID: "caf4036c-1f50-4772-b22a-152d2df0b640"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:10.581756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581711 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:10.581756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581750 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfgmh\" (UniqueName: \"kubernetes.io/projected/caf4036c-1f50-4772-b22a-152d2df0b640-kube-api-access-gfgmh\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:10.581756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581761 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-oauth-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:10.581756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581769 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-oauth-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:10.582010 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581778 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/caf4036c-1f50-4772-b22a-152d2df0b640-console-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:10.582010 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:10.581786 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/caf4036c-1f50-4772-b22a-152d2df0b640-console-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:54:11.213617 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.213584 2559 generic.go:356] "Generic (PLEG): container finished" podID="caf4036c-1f50-4772-b22a-152d2df0b640" containerID="983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c" exitCode=0 Aug 04 17:54:11.213799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.213632 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d76487d64-rxhn7" Aug 04 17:54:11.213799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.213677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d76487d64-rxhn7" event={"ID":"caf4036c-1f50-4772-b22a-152d2df0b640","Type":"ContainerDied","Data":"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c"} Aug 04 17:54:11.213799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.213729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d76487d64-rxhn7" event={"ID":"caf4036c-1f50-4772-b22a-152d2df0b640","Type":"ContainerDied","Data":"913fbf51d5951286326c2ebd04695c50fafa65d4faae4d8f8789494f2b4195d0"} Aug 04 17:54:11.213799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.213751 2559 scope.go:117] "RemoveContainer" containerID="983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c" Aug 04 17:54:11.221723 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.221630 2559 scope.go:117] "RemoveContainer" containerID="983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c" Aug 04 17:54:11.221987 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:54:11.221936 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c\": container with ID starting with 983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c not found: ID does not exist" containerID="983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c" Aug 04 17:54:11.221987 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.221960 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c"} err="failed to get container status \"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c\": rpc error: code = NotFound desc = could not find container \"983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c\": container with ID starting with 983b4a36286aecf18e1620abdcfde0538f28160e89a54eeae97a128aea18dc9c not found: ID does not exist" Aug 04 17:54:11.231037 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.231016 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:54:11.235550 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:11.235527 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d76487d64-rxhn7"] Aug 04 17:54:12.737624 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:12.737583 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf4036c-1f50-4772-b22a-152d2df0b640" path="/var/lib/kubelet/pods/caf4036c-1f50-4772-b22a-152d2df0b640/volumes" Aug 04 17:54:46.786858 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.786823 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f77f59b44-ldgvv"] Aug 04 17:54:46.787286 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.787074 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf4036c-1f50-4772-b22a-152d2df0b640" containerName="console" Aug 04 17:54:46.787286 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.787083 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf4036c-1f50-4772-b22a-152d2df0b640" containerName="console" Aug 04 17:54:46.787286 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.787127 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf4036c-1f50-4772-b22a-152d2df0b640" containerName="console" Aug 04 17:54:46.789123 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.789107 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.791553 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.791528 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 17:54:46.791676 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.791538 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 17:54:46.791676 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.791626 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 17:54:46.791676 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.791649 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 17:54:46.792187 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.792173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 17:54:46.792255 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.792243 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-m4pfn\"" Aug 04 17:54:46.798436 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.798416 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 17:54:46.804153 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.804135 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f77f59b44-ldgvv"] Aug 04 17:54:46.937997 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.937959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-serving-certs-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.937997 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.937990 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-metrics-client-ca\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfpz\" (UniqueName: \"kubernetes.io/projected/3bbee3bc-6ba1-4162-a850-b7d93684e62f-kube-api-access-rlfpz\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938146 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-federate-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:46.938200 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:46.938190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039164 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-serving-certs-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-metrics-client-ca\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039253 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfpz\" (UniqueName: \"kubernetes.io/projected/3bbee3bc-6ba1-4162-a850-b7d93684e62f-kube-api-access-rlfpz\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039282 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-federate-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.039602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.039396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.040122 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.040037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-metrics-client-ca\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.040382 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.040078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-serving-certs-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.040511 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.040240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.041905 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.041878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.042161 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.042139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-telemeter-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.042227 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.042154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-secret-telemeter-client\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.042227 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.042140 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bbee3bc-6ba1-4162-a850-b7d93684e62f-federate-client-tls\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.050821 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.050795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfpz\" (UniqueName: \"kubernetes.io/projected/3bbee3bc-6ba1-4162-a850-b7d93684e62f-kube-api-access-rlfpz\") pod \"telemeter-client-7f77f59b44-ldgvv\" (UID: \"3bbee3bc-6ba1-4162-a850-b7d93684e62f\") " pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.097982 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.097954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" Aug 04 17:54:47.237424 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.237396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f77f59b44-ldgvv"] Aug 04 17:54:47.240433 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:54:47.240402 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbee3bc_6ba1_4162_a850_b7d93684e62f.slice/crio-7d66800029dc5dc27b9d2f8c3daaa17e52791e22532e8b64c876261701206c40 WatchSource:0}: Error finding container 7d66800029dc5dc27b9d2f8c3daaa17e52791e22532e8b64c876261701206c40: Status 404 returned error can't find the container with id 7d66800029dc5dc27b9d2f8c3daaa17e52791e22532e8b64c876261701206c40 Aug 04 17:54:47.331215 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:47.331134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" event={"ID":"3bbee3bc-6ba1-4162-a850-b7d93684e62f","Type":"ContainerStarted","Data":"7d66800029dc5dc27b9d2f8c3daaa17e52791e22532e8b64c876261701206c40"} Aug 04 17:54:49.339057 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:49.338978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" event={"ID":"3bbee3bc-6ba1-4162-a850-b7d93684e62f","Type":"ContainerStarted","Data":"d6a9ab86e91519edfd7d26585e998e4d6382e11caaab0d3b5ac61ad881bd560e"} Aug 04 17:54:51.348536 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.348489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" event={"ID":"3bbee3bc-6ba1-4162-a850-b7d93684e62f","Type":"ContainerStarted","Data":"84dfba23dec70212b3c0742a3066a3f025a8afea83d7658a401a175f49eb2bbe"} Aug 04 17:54:51.348929 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.348556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" event={"ID":"3bbee3bc-6ba1-4162-a850-b7d93684e62f","Type":"ContainerStarted","Data":"4c2102487a871b5f5999fc4321596130a13c87cba4cb5d3ebec645f32559428e"} Aug 04 17:54:51.377436 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.377386 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f77f59b44-ldgvv" podStartSLOduration=2.23624385 podStartE2EDuration="5.377368958s" totalImagesPullingTime="3.141125108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:54:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:54:51.37492106 +0000 UTC m=+153.117075130" watchObservedRunningTime="2026-08-04 17:54:51.377368958 +0000 UTC m=+153.119523060" Aug 04 17:54:51.494669 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.494635 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:54:51.496554 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.496536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.512389 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.512367 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:54:51.578173 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtstp\" (UniqueName: \"kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578173 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.578419 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.578354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679552 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtstp\" (UniqueName: \"kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679552 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679482 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679552 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679535 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679552 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679854 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679854 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.679854 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.679707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.680315 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.680294 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.680414 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.680344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.680494 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.680478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.680883 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.680863 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.682039 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.682019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.682202 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.682182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.693710 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.693680 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtstp\" (UniqueName: \"kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp\") pod \"console-d77dbc4b5-rsrvk\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.805160 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.805126 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:54:51.933096 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:51.933014 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:54:51.936715 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:54:51.936690 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c1e2dd_43fa_4254_b2c1_2eaccb74d43b.slice/crio-81976ca1a03ed2e089f2b87f1dccdaea654051e8c6a42cb3d1822310e5acaa73 WatchSource:0}: Error finding container 81976ca1a03ed2e089f2b87f1dccdaea654051e8c6a42cb3d1822310e5acaa73: Status 404 returned error can't find the container with id 81976ca1a03ed2e089f2b87f1dccdaea654051e8c6a42cb3d1822310e5acaa73 Aug 04 17:54:52.353723 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:52.353682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77dbc4b5-rsrvk" event={"ID":"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b","Type":"ContainerStarted","Data":"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501"} Aug 04 17:54:52.354073 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:52.353766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77dbc4b5-rsrvk" event={"ID":"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b","Type":"ContainerStarted","Data":"81976ca1a03ed2e089f2b87f1dccdaea654051e8c6a42cb3d1822310e5acaa73"} Aug 04 17:54:52.373920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:54:52.373868 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d77dbc4b5-rsrvk" podStartSLOduration=1.373851695 podStartE2EDuration="1.373851695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:54:52.373304857 +0000 UTC m=+154.115458916" watchObservedRunningTime="2026-08-04 17:54:52.373851695 +0000 UTC m=+154.116005754" Aug 04 17:55:01.805833 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:01.805794 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:55:01.806256 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:01.805857 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:55:01.810753 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:01.810733 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:55:02.391202 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:02.391172 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:55:02.457225 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:02.457195 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:55:27.479299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.479237 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ff678c6d8-stn9p" podUID="e388a65e-d08b-4c8b-b17f-04f24f674473" containerName="console" containerID="cri-o://b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5" gracePeriod=15 Aug 04 17:55:27.718081 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.718055 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:55:27.758185 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758109 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758185 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758386 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758191 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758386 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758219 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758386 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758248 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758568 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758439 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758568 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758527 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqh45\" (UniqueName: \"kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45\") pod \"e388a65e-d08b-4c8b-b17f-04f24f674473\" (UID: \"e388a65e-d08b-4c8b-b17f-04f24f674473\") " Aug 04 17:55:27.758675 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758573 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:55:27.758675 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758624 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca" (OuterVolumeSpecName: "service-ca") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:55:27.758675 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758637 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:55:27.758812 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758778 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-trusted-ca-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.758812 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758799 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.758905 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.758815 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-oauth-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.759093 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.759042 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config" (OuterVolumeSpecName: "console-config") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:55:27.760430 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.760407 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:55:27.760626 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.760603 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45" (OuterVolumeSpecName: "kube-api-access-lqh45") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "kube-api-access-lqh45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:55:27.760626 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.760605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e388a65e-d08b-4c8b-b17f-04f24f674473" (UID: "e388a65e-d08b-4c8b-b17f-04f24f674473"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:55:27.859897 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.859851 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e388a65e-d08b-4c8b-b17f-04f24f674473-console-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.859897 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.859903 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-oauth-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.859897 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.859914 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e388a65e-d08b-4c8b-b17f-04f24f674473-console-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:27.860130 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:27.859922 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqh45\" (UniqueName: \"kubernetes.io/projected/e388a65e-d08b-4c8b-b17f-04f24f674473-kube-api-access-lqh45\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:55:28.479908 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.479872 2559 generic.go:356] "Generic (PLEG): container finished" podID="e388a65e-d08b-4c8b-b17f-04f24f674473" containerID="b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5" exitCode=0 Aug 04 17:55:28.480299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.479918 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff678c6d8-stn9p" Aug 04 17:55:28.480299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.479954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff678c6d8-stn9p" event={"ID":"e388a65e-d08b-4c8b-b17f-04f24f674473","Type":"ContainerDied","Data":"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5"} Aug 04 17:55:28.480299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.480001 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff678c6d8-stn9p" event={"ID":"e388a65e-d08b-4c8b-b17f-04f24f674473","Type":"ContainerDied","Data":"5355cbf92e8848a575d035a5e2676cab84eabfc7076e0c4d007ffb4671439643"} Aug 04 17:55:28.480299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.480022 2559 scope.go:117] "RemoveContainer" containerID="b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5" Aug 04 17:55:28.487996 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.487979 2559 scope.go:117] "RemoveContainer" containerID="b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5" Aug 04 17:55:28.488242 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:55:28.488224 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5\": container with ID starting with b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5 not found: ID does not exist" containerID="b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5" Aug 04 17:55:28.488284 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.488251 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5"} err="failed to get container status \"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5\": rpc error: code = NotFound desc = could not find container \"b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5\": container with ID starting with b67c43a09c0a8e08e720888a577f68262f6fc7435bc2e289e4542dfc448917e5 not found: ID does not exist" Aug 04 17:55:28.512699 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.512673 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:55:28.529822 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.529799 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ff678c6d8-stn9p"] Aug 04 17:55:28.737310 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:55:28.737236 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e388a65e-d08b-4c8b-b17f-04f24f674473" path="/var/lib/kubelet/pods/e388a65e-d08b-4c8b-b17f-04f24f674473/volumes" Aug 04 17:56:03.924971 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.924930 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:56:03.925571 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.925304 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e388a65e-d08b-4c8b-b17f-04f24f674473" containerName="console" Aug 04 17:56:03.925571 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.925322 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e388a65e-d08b-4c8b-b17f-04f24f674473" containerName="console" Aug 04 17:56:03.925571 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.925402 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e388a65e-d08b-4c8b-b17f-04f24f674473" containerName="console" Aug 04 17:56:03.928476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.928457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:03.944112 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:03.944089 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:56:04.023452 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023452 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfsz\" (UniqueName: \"kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.023686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.023669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124482 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124482 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124482 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfsz\" (UniqueName: \"kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.124665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.124607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.125276 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.125249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.125388 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.125257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.125466 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.125447 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.125578 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.125466 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.127137 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.127117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.127222 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.127207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.135901 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.135877 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfsz\" (UniqueName: \"kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz\") pod \"console-784f988487-5x99j\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.237610 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.237526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:04.381794 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.381765 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:56:04.384450 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:56:04.384425 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7706572c_efce_4273_8186_7b508a399459.slice/crio-5aa20e30e29c01e2d63998d79a3ee0118e83f7805ac1ff586a6b66cef0acc822 WatchSource:0}: Error finding container 5aa20e30e29c01e2d63998d79a3ee0118e83f7805ac1ff586a6b66cef0acc822: Status 404 returned error can't find the container with id 5aa20e30e29c01e2d63998d79a3ee0118e83f7805ac1ff586a6b66cef0acc822 Aug 04 17:56:04.603870 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.603839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784f988487-5x99j" event={"ID":"7706572c-efce-4273-8186-7b508a399459","Type":"ContainerStarted","Data":"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5"} Aug 04 17:56:04.604034 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.603881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784f988487-5x99j" event={"ID":"7706572c-efce-4273-8186-7b508a399459","Type":"ContainerStarted","Data":"5aa20e30e29c01e2d63998d79a3ee0118e83f7805ac1ff586a6b66cef0acc822"} Aug 04 17:56:04.638407 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:04.638363 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-784f988487-5x99j" podStartSLOduration=1.6383494010000001 podStartE2EDuration="1.638349401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:56:04.636605368 +0000 UTC m=+226.378759426" watchObservedRunningTime="2026-08-04 17:56:04.638349401 +0000 UTC m=+226.380503458" Aug 04 17:56:14.238463 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:14.238410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:14.238929 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:14.238487 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:14.242968 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:14.242946 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:14.642694 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:14.642667 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:56:14.699881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:14.699852 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:56:39.719820 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:39.719717 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d77dbc4b5-rsrvk" podUID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" containerName="console" containerID="cri-o://43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501" gracePeriod=15 Aug 04 17:56:39.955760 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:39.955738 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:56:40.008638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008533 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008609 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008852 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008663 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtstp\" (UniqueName: \"kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008852 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008685 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008852 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008733 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008852 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.008852 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008784 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert\") pod \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\" (UID: \"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b\") " Aug 04 17:56:40.009089 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.008898 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config" (OuterVolumeSpecName: "console-config") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:56:40.009146 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.009121 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.009202 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.009146 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca" (OuterVolumeSpecName: "service-ca") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:56:40.009202 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.009170 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:56:40.009288 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.009230 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:56:40.010888 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.010849 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:56:40.011005 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.010908 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp" (OuterVolumeSpecName: "kube-api-access-jtstp") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "kube-api-access-jtstp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:56:40.011005 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.010906 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" (UID: "30c1e2dd-43fa-4254-b2c1-2eaccb74d43b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:56:40.109755 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109722 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-oauth-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.109755 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109750 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtstp\" (UniqueName: \"kubernetes.io/projected/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-kube-api-access-jtstp\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.109755 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109759 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-trusted-ca-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.109755 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109769 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-console-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.110009 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109778 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.110009 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.109785 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b-oauth-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:56:40.727041 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.727010 2559 generic.go:356] "Generic (PLEG): container finished" podID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" containerID="43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501" exitCode=0 Aug 04 17:56:40.727483 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.727051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d77dbc4b5-rsrvk" Aug 04 17:56:40.727483 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.727072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77dbc4b5-rsrvk" event={"ID":"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b","Type":"ContainerDied","Data":"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501"} Aug 04 17:56:40.727483 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.727112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d77dbc4b5-rsrvk" event={"ID":"30c1e2dd-43fa-4254-b2c1-2eaccb74d43b","Type":"ContainerDied","Data":"81976ca1a03ed2e089f2b87f1dccdaea654051e8c6a42cb3d1822310e5acaa73"} Aug 04 17:56:40.727483 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.727133 2559 scope.go:117] "RemoveContainer" containerID="43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501" Aug 04 17:56:40.736019 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.735998 2559 scope.go:117] "RemoveContainer" containerID="43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501" Aug 04 17:56:40.736284 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:56:40.736263 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501\": container with ID starting with 43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501 not found: ID does not exist" containerID="43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501" Aug 04 17:56:40.736364 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.736291 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501"} err="failed to get container status \"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501\": rpc error: code = NotFound desc = could not find container \"43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501\": container with ID starting with 43e99cb7ab9f004504522d8b967259ee243a0dd4bc950010586690b688d37501 not found: ID does not exist" Aug 04 17:56:40.751929 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.751910 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:56:40.755570 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:40.755550 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d77dbc4b5-rsrvk"] Aug 04 17:56:42.737172 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:42.737140 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" path="/var/lib/kubelet/pods/30c1e2dd-43fa-4254-b2c1-2eaccb74d43b/volumes" Aug 04 17:56:57.807566 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.807530 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz"] Aug 04 17:56:57.807954 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.807782 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" containerName="console" Aug 04 17:56:57.807954 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.807793 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" containerName="console" Aug 04 17:56:57.807954 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.807845 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1e2dd-43fa-4254-b2c1-2eaccb74d43b" containerName="console" Aug 04 17:56:57.812267 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.812249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:57.814943 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.814928 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4l8z\"" Aug 04 17:56:57.815080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.815062 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 17:56:57.815647 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.815630 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 17:56:57.823655 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.823625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz"] Aug 04 17:56:57.938555 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.938522 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:57.938555 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.938551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:57.938759 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:57.938577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szv9\" (UniqueName: \"kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.040020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.039986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.040020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.040021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.040241 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.040049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szv9\" (UniqueName: \"kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.040431 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.040413 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.040469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.040434 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.050683 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.050657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szv9\" (UniqueName: \"kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.120658 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.120579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:56:58.248624 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.248586 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz"] Aug 04 17:56:58.252361 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:56:58.252333 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5dacf8_bbae_4e54_85a3_2134a693f28b.slice/crio-13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf WatchSource:0}: Error finding container 13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf: Status 404 returned error can't find the container with id 13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf Aug 04 17:56:58.787571 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:56:58.787532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" event={"ID":"8a5dacf8-bbae-4e54-85a3-2134a693f28b","Type":"ContainerStarted","Data":"13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf"} Aug 04 17:57:04.814584 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:04.814550 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerID="12e094cce1fd83c1e4a1f5d8691e617df5d4c6df0f9ce46d69d4b3f69b6c6454" exitCode=0 Aug 04 17:57:04.815062 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:04.814615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" event={"ID":"8a5dacf8-bbae-4e54-85a3-2134a693f28b","Type":"ContainerDied","Data":"12e094cce1fd83c1e4a1f5d8691e617df5d4c6df0f9ce46d69d4b3f69b6c6454"} Aug 04 17:57:07.826559 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:07.826518 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerID="3fc6f3e5f7114e2a55d606a8afaaf6c671a107f4290311a67df34f5f8d206d13" exitCode=0 Aug 04 17:57:07.826559 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:07.826534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" event={"ID":"8a5dacf8-bbae-4e54-85a3-2134a693f28b","Type":"ContainerDied","Data":"3fc6f3e5f7114e2a55d606a8afaaf6c671a107f4290311a67df34f5f8d206d13"} Aug 04 17:57:14.855275 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:14.855240 2559 generic.go:356] "Generic (PLEG): container finished" podID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerID="00a1d590bf5fc127e5f1bdf19f79882d23b59442ee4339d446da986c2e8ccb72" exitCode=0 Aug 04 17:57:14.855672 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:14.855291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" event={"ID":"8a5dacf8-bbae-4e54-85a3-2134a693f28b","Type":"ContainerDied","Data":"00a1d590bf5fc127e5f1bdf19f79882d23b59442ee4339d446da986c2e8ccb72"} Aug 04 17:57:15.982324 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:15.982300 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:57:16.097919 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.097882 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util\") pod \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " Aug 04 17:57:16.098080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.097929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle\") pod \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " Aug 04 17:57:16.098080 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.098000 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9szv9\" (UniqueName: \"kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9\") pod \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\" (UID: \"8a5dacf8-bbae-4e54-85a3-2134a693f28b\") " Aug 04 17:57:16.098568 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.098539 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle" (OuterVolumeSpecName: "bundle") pod "8a5dacf8-bbae-4e54-85a3-2134a693f28b" (UID: "8a5dacf8-bbae-4e54-85a3-2134a693f28b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:16.100271 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.100237 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9" (OuterVolumeSpecName: "kube-api-access-9szv9") pod "8a5dacf8-bbae-4e54-85a3-2134a693f28b" (UID: "8a5dacf8-bbae-4e54-85a3-2134a693f28b"). InnerVolumeSpecName "kube-api-access-9szv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:57:16.102057 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.102034 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util" (OuterVolumeSpecName: "util") pod "8a5dacf8-bbae-4e54-85a3-2134a693f28b" (UID: "8a5dacf8-bbae-4e54-85a3-2134a693f28b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:16.198872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.198773 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9szv9\" (UniqueName: \"kubernetes.io/projected/8a5dacf8-bbae-4e54-85a3-2134a693f28b-kube-api-access-9szv9\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:16.198872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.198817 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:16.198872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.198827 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5dacf8-bbae-4e54-85a3-2134a693f28b-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:16.863726 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.863703 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" Aug 04 17:57:16.863895 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.863696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540w5lwz" event={"ID":"8a5dacf8-bbae-4e54-85a3-2134a693f28b","Type":"ContainerDied","Data":"13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf"} Aug 04 17:57:16.863895 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:16.863829 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13e49b4e19aad7c44d4d6897f37028db48d99cadef9bc27243f2fc9a79fc44cf" Aug 04 17:57:18.627699 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:18.627669 2559 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.843481 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb"] Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844017 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="extract" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844034 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="extract" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844065 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="pull" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844074 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="pull" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844089 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="util" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844100 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="util" Aug 04 17:57:20.845402 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.844205 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5dacf8-bbae-4e54-85a3-2134a693f28b" containerName="extract" Aug 04 17:57:20.846816 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.846625 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:20.849697 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.849676 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:57:20.849820 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.849732 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 17:57:20.849994 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.849978 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nrmph\"" Aug 04 17:57:20.861999 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.861976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb"] Aug 04 17:57:20.933584 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.933551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/96aff547-3f5c-4bdb-94b3-5243530da1ca-kube-api-access-wvrxk\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:20.933584 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:20.933589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96aff547-3f5c-4bdb-94b3-5243530da1ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.034631 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.034594 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/96aff547-3f5c-4bdb-94b3-5243530da1ca-kube-api-access-wvrxk\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.034631 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.034631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96aff547-3f5c-4bdb-94b3-5243530da1ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.035004 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.034988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96aff547-3f5c-4bdb-94b3-5243530da1ca-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.044090 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.044065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrxk\" (UniqueName: \"kubernetes.io/projected/96aff547-3f5c-4bdb-94b3-5243530da1ca-kube-api-access-wvrxk\") pod \"cert-manager-operator-controller-manager-55c444bf86-p5wrb\" (UID: \"96aff547-3f5c-4bdb-94b3-5243530da1ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.155596 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.155493 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" Aug 04 17:57:21.317580 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.317554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb"] Aug 04 17:57:21.319275 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:57:21.319251 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96aff547_3f5c_4bdb_94b3_5243530da1ca.slice/crio-7ebe1de2a1edcd1269cfb21fe5572680bb6a37184f28f5e6e0833d7de6e60581 WatchSource:0}: Error finding container 7ebe1de2a1edcd1269cfb21fe5572680bb6a37184f28f5e6e0833d7de6e60581: Status 404 returned error can't find the container with id 7ebe1de2a1edcd1269cfb21fe5572680bb6a37184f28f5e6e0833d7de6e60581 Aug 04 17:57:21.321975 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.321958 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 17:57:21.883051 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:21.883016 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" event={"ID":"96aff547-3f5c-4bdb-94b3-5243530da1ca","Type":"ContainerStarted","Data":"7ebe1de2a1edcd1269cfb21fe5572680bb6a37184f28f5e6e0833d7de6e60581"} Aug 04 17:57:23.893209 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:23.893169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" event={"ID":"96aff547-3f5c-4bdb-94b3-5243530da1ca","Type":"ContainerStarted","Data":"11df51847beec81e8a215311f719fa6d5186e264b6e6cd1682dbd43d7532370b"} Aug 04 17:57:23.918313 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:23.918259 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p5wrb" podStartSLOduration=2.349354456 podStartE2EDuration="3.918244506s" totalImagesPullingTime="1.56889005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:57:23.916464646 +0000 UTC m=+305.658618717" watchObservedRunningTime="2026-08-04 17:57:23.918244506 +0000 UTC m=+305.660398591" Aug 04 17:57:24.847009 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.846970 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6"] Aug 04 17:57:24.849241 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.849222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:24.852154 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.852136 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4l8z\"" Aug 04 17:57:24.852469 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.852455 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 17:57:24.852665 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.852651 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 17:57:24.866437 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.866417 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6"] Aug 04 17:57:24.967359 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.967326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:24.967746 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.967385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:24.967746 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:24.967415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kksgl\" (UniqueName: \"kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.067881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.067847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.068075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.067900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kksgl\" (UniqueName: \"kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.068075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.067946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.068272 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.068251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.068349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.068271 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.078101 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.078079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kksgl\" (UniqueName: \"kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.157952 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.157872 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:25.300040 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.300011 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6"] Aug 04 17:57:25.650551 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.650512 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nfkxd"] Aug 04 17:57:25.652564 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.652545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.655350 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.655327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 17:57:25.655466 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.655369 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-n8869\"" Aug 04 17:57:25.655466 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.655334 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 17:57:25.665368 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.665346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nfkxd"] Aug 04 17:57:25.774506 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.774471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfb5n\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-kube-api-access-sfb5n\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.774692 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.774561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.875599 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.875567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfb5n\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-kube-api-access-sfb5n\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.875888 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.875859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.891448 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.891410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.892019 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.892000 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfb5n\" (UniqueName: \"kubernetes.io/projected/696b1b13-f07b-4ac4-94e8-71b44d35d599-kube-api-access-sfb5n\") pod \"cert-manager-webhook-6486ffd997-nfkxd\" (UID: \"696b1b13-f07b-4ac4-94e8-71b44d35d599\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:25.902670 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.902594 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerID="5cc83c0a83702c3e525b9e9dcdb4c7fa976c1fa8c737f5dc9456d65f848133a8" exitCode=0 Aug 04 17:57:25.902670 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.902656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" event={"ID":"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e","Type":"ContainerDied","Data":"5cc83c0a83702c3e525b9e9dcdb4c7fa976c1fa8c737f5dc9456d65f848133a8"} Aug 04 17:57:25.902855 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.902695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" event={"ID":"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e","Type":"ContainerStarted","Data":"6676e6fad4a545a611ccad924d8bdb4afe573c8a58cecad559876700b4d91d72"} Aug 04 17:57:25.972407 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:25.972368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:26.103577 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:26.103553 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nfkxd"] Aug 04 17:57:26.106087 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:57:26.106057 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod696b1b13_f07b_4ac4_94e8_71b44d35d599.slice/crio-5d0b51ca26d5a8ace7287e6049fc584c6fefff4a25f6bb0224bcdffe4dca6b7c WatchSource:0}: Error finding container 5d0b51ca26d5a8ace7287e6049fc584c6fefff4a25f6bb0224bcdffe4dca6b7c: Status 404 returned error can't find the container with id 5d0b51ca26d5a8ace7287e6049fc584c6fefff4a25f6bb0224bcdffe4dca6b7c Aug 04 17:57:26.913179 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:26.913137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" event={"ID":"696b1b13-f07b-4ac4-94e8-71b44d35d599","Type":"ContainerStarted","Data":"5d0b51ca26d5a8ace7287e6049fc584c6fefff4a25f6bb0224bcdffe4dca6b7c"} Aug 04 17:57:28.922208 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:28.922179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" event={"ID":"696b1b13-f07b-4ac4-94e8-71b44d35d599","Type":"ContainerStarted","Data":"7e15fcc8c62b6f630aebfb7e4fcef5a44f773c867c93de82da513dfd48a12b3d"} Aug 04 17:57:28.922624 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:28.922231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:28.924003 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:28.923966 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerID="fd1c41b1f7ed7b3a5cd404e5807729b1a31aa226a13d5198ccb47ab5f85a6f78" exitCode=0 Aug 04 17:57:28.924135 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:28.924030 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" event={"ID":"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e","Type":"ContainerDied","Data":"fd1c41b1f7ed7b3a5cd404e5807729b1a31aa226a13d5198ccb47ab5f85a6f78"} Aug 04 17:57:28.942477 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:28.942418 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" podStartSLOduration=1.233180191 podStartE2EDuration="3.942399476s" totalImagesPullingTime="2.709219285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:57:28.940024945 +0000 UTC m=+310.682179006" watchObservedRunningTime="2026-08-04 17:57:28.942399476 +0000 UTC m=+310.684553534" Aug 04 17:57:29.930086 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:29.930051 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerID="a6e3d7a26b564552d6842b97a4c575a33a0947b898ad79e1f78a5b005ea8d9f0" exitCode=0 Aug 04 17:57:29.930445 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:29.930134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" event={"ID":"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e","Type":"ContainerDied","Data":"a6e3d7a26b564552d6842b97a4c575a33a0947b898ad79e1f78a5b005ea8d9f0"} Aug 04 17:57:31.055169 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.055145 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:31.221282 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.221170 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util\") pod \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " Aug 04 17:57:31.221282 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.221231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle\") pod \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " Aug 04 17:57:31.221551 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.221309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kksgl\" (UniqueName: \"kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl\") pod \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\" (UID: \"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e\") " Aug 04 17:57:31.221697 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.221671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle" (OuterVolumeSpecName: "bundle") pod "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" (UID: "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:31.223472 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.223439 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl" (OuterVolumeSpecName: "kube-api-access-kksgl") pod "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" (UID: "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e"). InnerVolumeSpecName "kube-api-access-kksgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:57:31.225790 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.225757 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util" (OuterVolumeSpecName: "util") pod "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" (UID: "d4e6517c-6b4b-4f22-b203-80d9ffcfb87e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:31.322457 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.322424 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kksgl\" (UniqueName: \"kubernetes.io/projected/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-kube-api-access-kksgl\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:31.322457 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.322451 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:31.322457 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.322463 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4e6517c-6b4b-4f22-b203-80d9ffcfb87e-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:31.939596 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.939562 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" event={"ID":"d4e6517c-6b4b-4f22-b203-80d9ffcfb87e","Type":"ContainerDied","Data":"6676e6fad4a545a611ccad924d8bdb4afe573c8a58cecad559876700b4d91d72"} Aug 04 17:57:31.939596 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.939600 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2bkc6" Aug 04 17:57:31.939798 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:31.939606 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6676e6fad4a545a611ccad924d8bdb4afe573c8a58cecad559876700b4d91d72" Aug 04 17:57:34.932601 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:34.932574 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nfkxd" Aug 04 17:57:37.268774 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.268740 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tj92l"] Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.268998 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="pull" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269007 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="pull" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269019 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="extract" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269024 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="extract" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269032 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="util" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269038 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="util" Aug 04 17:57:37.269216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.269076 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e6517c-6b4b-4f22-b203-80d9ffcfb87e" containerName="extract" Aug 04 17:57:37.272017 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.272003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.274154 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:57:37.274122 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"cert-manager-55b4d954d5-tj92l\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="3b568f41-f33d-4c2f-9b51-7d147a0b822e" pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.274564 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:57:37.274542 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cert-manager-dockercfg-29v7k\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29v7k\"" type="*v1.Secret" Aug 04 17:57:37.290682 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.290660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tj92l"] Aug 04 17:57:37.367446 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.367411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-bound-sa-token\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.367446 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.367445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qdk\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-kube-api-access-d8qdk\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.468421 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.468387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-bound-sa-token\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.468421 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.468421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qdk\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-kube-api-access-d8qdk\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.481796 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.481770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-bound-sa-token\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:37.484823 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:37.484804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qdk\" (UniqueName: \"kubernetes.io/projected/3b568f41-f33d-4c2f-9b51-7d147a0b822e-kube-api-access-d8qdk\") pod \"cert-manager-55b4d954d5-tj92l\" (UID: \"3b568f41-f33d-4c2f-9b51-7d147a0b822e\") " pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:38.559857 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.559825 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx"] Aug 04 17:57:38.562843 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.562828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.567795 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.567778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 17:57:38.568452 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.568437 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-nrghg\"" Aug 04 17:57:38.568525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.568453 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:57:38.579237 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.579215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6ht\" (UniqueName: \"kubernetes.io/projected/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-kube-api-access-jr6ht\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.579346 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.579279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.580841 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.580822 2559 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="cert-manager/cert-manager-55b4d954d5-tj92l" secret="" err="failed to sync secret cache: timed out waiting for the condition" Aug 04 17:57:38.580916 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.580863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tj92l" Aug 04 17:57:38.580916 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.580824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx"] Aug 04 17:57:38.680104 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.680076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6ht\" (UniqueName: \"kubernetes.io/projected/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-kube-api-access-jr6ht\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.680256 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.680123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.680451 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.680435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.693553 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.693526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6ht\" (UniqueName: \"kubernetes.io/projected/4e2e8c1d-cd51-4b61-befc-f22433a0c5ac-kube-api-access-jr6ht\") pod \"openshift-lws-operator-fd8ccff4c-v8dbx\" (UID: \"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.722823 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.722794 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tj92l"] Aug 04 17:57:38.725406 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.725390 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29v7k\"" Aug 04 17:57:38.725722 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:57:38.725702 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b568f41_f33d_4c2f_9b51_7d147a0b822e.slice/crio-1a678de6ab798b3e8b5922d3dc9a2033511fb224b928cc34d49d485cfe6eadd9 WatchSource:0}: Error finding container 1a678de6ab798b3e8b5922d3dc9a2033511fb224b928cc34d49d485cfe6eadd9: Status 404 returned error can't find the container with id 1a678de6ab798b3e8b5922d3dc9a2033511fb224b928cc34d49d485cfe6eadd9 Aug 04 17:57:38.871249 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.871172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" Aug 04 17:57:38.969261 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.969221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tj92l" event={"ID":"3b568f41-f33d-4c2f-9b51-7d147a0b822e","Type":"ContainerStarted","Data":"0718d13bc07c1d31ccad7db8d2c2e5c0bc6837c01cb83aba836e7dbe887af66d"} Aug 04 17:57:38.969412 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:38.969276 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tj92l" event={"ID":"3b568f41-f33d-4c2f-9b51-7d147a0b822e","Type":"ContainerStarted","Data":"1a678de6ab798b3e8b5922d3dc9a2033511fb224b928cc34d49d485cfe6eadd9"} Aug 04 17:57:39.003378 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:39.003324 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tj92l" podStartSLOduration=2.003308939 podStartE2EDuration="2.003308939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:57:39.000767964 +0000 UTC m=+320.742922040" watchObservedRunningTime="2026-08-04 17:57:39.003308939 +0000 UTC m=+320.745462996" Aug 04 17:57:39.011947 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:39.011924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx"] Aug 04 17:57:39.014683 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:57:39.014661 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e2e8c1d_cd51_4b61_befc_f22433a0c5ac.slice/crio-d0bc7d9391332e29a58d56b3defd5fbbf7935bf9721abcba31b0787cd06c44f0 WatchSource:0}: Error finding container d0bc7d9391332e29a58d56b3defd5fbbf7935bf9721abcba31b0787cd06c44f0: Status 404 returned error can't find the container with id d0bc7d9391332e29a58d56b3defd5fbbf7935bf9721abcba31b0787cd06c44f0 Aug 04 17:57:39.974523 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:39.974458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" event={"ID":"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac","Type":"ContainerStarted","Data":"d0bc7d9391332e29a58d56b3defd5fbbf7935bf9721abcba31b0787cd06c44f0"} Aug 04 17:57:41.985287 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:41.985253 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" event={"ID":"4e2e8c1d-cd51-4b61-befc-f22433a0c5ac","Type":"ContainerStarted","Data":"80fb4425b29a855be11143ae2f00a0d7e069e229da955b3384c4c8a1e50ca8eb"} Aug 04 17:57:42.006397 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:42.006354 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v8dbx" podStartSLOduration=1.935370713 podStartE2EDuration="4.006339198s" totalImagesPullingTime="2.070968485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:57:42.005250086 +0000 UTC m=+323.747404146" watchObservedRunningTime="2026-08-04 17:57:42.006339198 +0000 UTC m=+323.748493254" Aug 04 17:57:44.752539 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.752493 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj"] Aug 04 17:57:44.755848 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.755830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.760074 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.760047 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 17:57:44.760183 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.760050 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 17:57:44.760183 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.760049 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4l8z\"" Aug 04 17:57:44.782084 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.782063 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj"] Aug 04 17:57:44.830930 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.830897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9xv\" (UniqueName: \"kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.831092 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.830941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.831092 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.830987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.931407 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.931371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.931407 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.931416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.931695 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.931541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9xv\" (UniqueName: \"kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.931829 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.931809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.931868 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.931822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:44.940372 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:44.940350 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9xv\" (UniqueName: \"kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:45.064232 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:45.064196 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:45.203724 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:45.203695 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj"] Aug 04 17:57:45.205900 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:57:45.205869 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddfebb2_e74f_4be8_b991_f219b9569faa.slice/crio-ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8 WatchSource:0}: Error finding container ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8: Status 404 returned error can't find the container with id ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8 Aug 04 17:57:46.006771 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:46.006737 2559 generic.go:356] "Generic (PLEG): container finished" podID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerID="c53c5bb5b23d0a4ef08b0fc8e01f5d81cdf780422bbf86cee6c1a7fd00d8dde6" exitCode=0 Aug 04 17:57:46.007122 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:46.006802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" event={"ID":"2ddfebb2-e74f-4be8-b991-f219b9569faa","Type":"ContainerDied","Data":"c53c5bb5b23d0a4ef08b0fc8e01f5d81cdf780422bbf86cee6c1a7fd00d8dde6"} Aug 04 17:57:46.007122 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:46.006851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" event={"ID":"2ddfebb2-e74f-4be8-b991-f219b9569faa","Type":"ContainerStarted","Data":"ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8"} Aug 04 17:57:48.017654 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:48.017619 2559 generic.go:356] "Generic (PLEG): container finished" podID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerID="a5c3d6488684191199eeb12a5f82e0aa8d85db4fb940a89f13195d06478b9a7c" exitCode=0 Aug 04 17:57:48.018210 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:48.017664 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" event={"ID":"2ddfebb2-e74f-4be8-b991-f219b9569faa","Type":"ContainerDied","Data":"a5c3d6488684191199eeb12a5f82e0aa8d85db4fb940a89f13195d06478b9a7c"} Aug 04 17:57:49.024028 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:49.023989 2559 generic.go:356] "Generic (PLEG): container finished" podID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerID="0f28e96282e1e2c35e276e304bc2a3af5b3cd0ad615cd4bb62f346bf990c91c4" exitCode=0 Aug 04 17:57:49.024413 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:49.024074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" event={"ID":"2ddfebb2-e74f-4be8-b991-f219b9569faa","Type":"ContainerDied","Data":"0f28e96282e1e2c35e276e304bc2a3af5b3cd0ad615cd4bb62f346bf990c91c4"} Aug 04 17:57:50.146493 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.146470 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:50.172941 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.172912 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle\") pod \"2ddfebb2-e74f-4be8-b991-f219b9569faa\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " Aug 04 17:57:50.173091 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.172969 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv9xv\" (UniqueName: \"kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv\") pod \"2ddfebb2-e74f-4be8-b991-f219b9569faa\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " Aug 04 17:57:50.173091 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.173005 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util\") pod \"2ddfebb2-e74f-4be8-b991-f219b9569faa\" (UID: \"2ddfebb2-e74f-4be8-b991-f219b9569faa\") " Aug 04 17:57:50.174006 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.173976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle" (OuterVolumeSpecName: "bundle") pod "2ddfebb2-e74f-4be8-b991-f219b9569faa" (UID: "2ddfebb2-e74f-4be8-b991-f219b9569faa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:50.175242 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.175207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv" (OuterVolumeSpecName: "kube-api-access-sv9xv") pod "2ddfebb2-e74f-4be8-b991-f219b9569faa" (UID: "2ddfebb2-e74f-4be8-b991-f219b9569faa"). InnerVolumeSpecName "kube-api-access-sv9xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:57:50.178954 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.178833 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util" (OuterVolumeSpecName: "util") pod "2ddfebb2-e74f-4be8-b991-f219b9569faa" (UID: "2ddfebb2-e74f-4be8-b991-f219b9569faa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:57:50.274576 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.274546 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:50.274576 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.274572 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9xv\" (UniqueName: \"kubernetes.io/projected/2ddfebb2-e74f-4be8-b991-f219b9569faa-kube-api-access-sv9xv\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:50.274576 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:50.274583 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ddfebb2-e74f-4be8-b991-f219b9569faa-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:57:51.034127 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:51.034099 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" event={"ID":"2ddfebb2-e74f-4be8-b991-f219b9569faa","Type":"ContainerDied","Data":"ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8"} Aug 04 17:57:51.034272 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:51.034130 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs5bpj" Aug 04 17:57:51.034272 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:57:51.034145 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee75f2fe01a4804efc55b975ddd00f8b5d27a998baf4a3bebce616f7634889b8" Aug 04 17:58:01.570945 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.570912 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r"] Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571165 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="pull" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571174 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="pull" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571188 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="util" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571194 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="util" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571207 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="extract" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571212 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="extract" Aug 04 17:58:01.571297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.571259 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ddfebb2-e74f-4be8-b991-f219b9569faa" containerName="extract" Aug 04 17:58:01.575260 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.575243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.587359 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.587341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 17:58:01.588067 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.588053 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 17:58:01.615767 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.615746 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 17:58:01.615872 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.615857 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w7vjj\"" Aug 04 17:58:01.615917 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.615894 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 17:58:01.636639 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.636613 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r"] Aug 04 17:58:01.673282 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.673248 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.673440 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.673289 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.673440 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.673325 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgl2w\" (UniqueName: \"kubernetes.io/projected/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-kube-api-access-rgl2w\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.673440 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.673408 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.673580 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.673448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774382 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774382 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774662 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgl2w\" (UniqueName: \"kubernetes.io/projected/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-kube-api-access-rgl2w\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774662 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774662 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.774810 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.774766 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.776776 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.776747 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.777112 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.777092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.777155 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.777098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.799529 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.799485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgl2w\" (UniqueName: \"kubernetes.io/projected/6a4a2a8d-30ab-4c39-9a0f-47ca8c800709-kube-api-access-rgl2w\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r\" (UID: \"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:01.884344 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:01.884253 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:02.022599 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:02.022574 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r"] Aug 04 17:58:02.024395 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:02.024372 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4a2a8d_30ab_4c39_9a0f_47ca8c800709.slice/crio-ed7e70276951f350a126903cda419bab4b8731e0cb07e1ee496a7279db156adf WatchSource:0}: Error finding container ed7e70276951f350a126903cda419bab4b8731e0cb07e1ee496a7279db156adf: Status 404 returned error can't find the container with id ed7e70276951f350a126903cda419bab4b8731e0cb07e1ee496a7279db156adf Aug 04 17:58:02.080750 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:02.080712 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" event={"ID":"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709","Type":"ContainerStarted","Data":"ed7e70276951f350a126903cda419bab4b8731e0cb07e1ee496a7279db156adf"} Aug 04 17:58:06.102157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:06.102117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" event={"ID":"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709","Type":"ContainerStarted","Data":"5b033d25f744ff98f7f5f3f69308122cbd6ae2fd92a847f66bdebe7475310acc"} Aug 04 17:58:07.108333 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:07.108301 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a4a2a8d-30ab-4c39-9a0f-47ca8c800709" containerID="5b033d25f744ff98f7f5f3f69308122cbd6ae2fd92a847f66bdebe7475310acc" exitCode=0 Aug 04 17:58:07.108333 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:07.108338 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" event={"ID":"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709","Type":"ContainerDied","Data":"5b033d25f744ff98f7f5f3f69308122cbd6ae2fd92a847f66bdebe7475310acc"} Aug 04 17:58:08.114702 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:08.114668 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" event={"ID":"6a4a2a8d-30ab-4c39-9a0f-47ca8c800709","Type":"ContainerStarted","Data":"a4d2ed58f481fe14e90d3707e8b3ac8962c3c53a7f22ae778e5ca0150eb1da88"} Aug 04 17:58:08.115049 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:08.114779 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:08.140232 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:08.140176 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" podStartSLOduration=2.662559666 podStartE2EDuration="7.140160383s" totalImagesPullingTime="3.538448777s" totalInitContainerRuntime="939.15194ms" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:08.139453714 +0000 UTC m=+349.881607771" watchObservedRunningTime="2026-08-04 17:58:08.140160383 +0000 UTC m=+349.882314441" Aug 04 17:58:19.120856 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:19.120783 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r" Aug 04 17:58:21.715961 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.715921 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-jhv6z"] Aug 04 17:58:21.724923 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.724890 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pldx4"] Aug 04 17:58:21.725068 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.725047 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.728185 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.728167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.730194 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.730175 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 17:58:21.730375 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.730357 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 17:58:21.730782 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.730763 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wm2t2\"" Aug 04 17:58:21.730947 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.730921 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 17:58:21.731047 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.731012 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 17:58:21.731157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.731139 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 17:58:21.731307 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.731281 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-td6qp\"" Aug 04 17:58:21.739487 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.739463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-jhv6z"] Aug 04 17:58:21.741270 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.741244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.741380 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.741290 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f2c87a-30c0-4652-a9a5-80baf5933464-tls-certs\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.741380 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.741332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jn7n\" (UniqueName: \"kubernetes.io/projected/9fae65b2-3848-43fb-bf33-b9756d8e3d08-kube-api-access-8jn7n\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.741380 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.741355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79f2c87a-30c0-4652-a9a5-80baf5933464-tmp\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.741575 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.741451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxk7x\" (UniqueName: \"kubernetes.io/projected/79f2c87a-30c0-4652-a9a5-80baf5933464-kube-api-access-rxk7x\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.744076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.744015 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pldx4"] Aug 04 17:58:21.841870 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.841835 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxk7x\" (UniqueName: \"kubernetes.io/projected/79f2c87a-30c0-4652-a9a5-80baf5933464-kube-api-access-rxk7x\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.841870 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.841873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.842126 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.841899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f2c87a-30c0-4652-a9a5-80baf5933464-tls-certs\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.842126 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.841930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jn7n\" (UniqueName: \"kubernetes.io/projected/9fae65b2-3848-43fb-bf33-b9756d8e3d08-kube-api-access-8jn7n\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.842126 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.841955 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79f2c87a-30c0-4652-a9a5-80baf5933464-tmp\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.842126 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:58:21.842046 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 17:58:21.842126 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:58:21.842126 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert podName:9fae65b2-3848-43fb-bf33-b9756d8e3d08 nodeName:}" failed. No retries permitted until 2026-08-04 17:58:22.34210401 +0000 UTC m=+364.084258050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert") pod "odh-model-controller-7998c67dc4-pldx4" (UID: "9fae65b2-3848-43fb-bf33-b9756d8e3d08") : secret "odh-model-controller-webhook-cert" not found Aug 04 17:58:21.844430 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.844401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79f2c87a-30c0-4652-a9a5-80baf5933464-tmp\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.844712 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.844681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79f2c87a-30c0-4652-a9a5-80baf5933464-tls-certs\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:21.863371 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.863344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jn7n\" (UniqueName: \"kubernetes.io/projected/9fae65b2-3848-43fb-bf33-b9756d8e3d08-kube-api-access-8jn7n\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:21.869283 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:21.869261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxk7x\" (UniqueName: \"kubernetes.io/projected/79f2c87a-30c0-4652-a9a5-80baf5933464-kube-api-access-rxk7x\") pod \"kube-auth-proxy-798d66f497-jhv6z\" (UID: \"79f2c87a-30c0-4652-a9a5-80baf5933464\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:22.036833 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.036807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" Aug 04 17:58:22.168725 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:22.168700 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f2c87a_30c0_4652_a9a5_80baf5933464.slice/crio-cdb01394a852185af34902fcf90a7e75b8892b798473002ffd0a5a669952ed57 WatchSource:0}: Error finding container cdb01394a852185af34902fcf90a7e75b8892b798473002ffd0a5a669952ed57: Status 404 returned error can't find the container with id cdb01394a852185af34902fcf90a7e75b8892b798473002ffd0a5a669952ed57 Aug 04 17:58:22.169996 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.169969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-jhv6z"] Aug 04 17:58:22.180422 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.180400 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" event={"ID":"79f2c87a-30c0-4652-a9a5-80baf5933464","Type":"ContainerStarted","Data":"cdb01394a852185af34902fcf90a7e75b8892b798473002ffd0a5a669952ed57"} Aug 04 17:58:22.346897 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.346820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:22.349227 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.349201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fae65b2-3848-43fb-bf33-b9756d8e3d08-cert\") pod \"odh-model-controller-7998c67dc4-pldx4\" (UID: \"9fae65b2-3848-43fb-bf33-b9756d8e3d08\") " pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:22.645232 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.645153 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:22.733310 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.733271 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l666w"] Aug 04 17:58:22.737216 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.737194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.746417 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.746378 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 17:58:22.750124 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.750099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wwfm\" (UniqueName: \"kubernetes.io/projected/2ca8fc11-9ec3-4bef-825d-d1ce99693438-kube-api-access-9wwfm\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.751353 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.750161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8fc11-9ec3-4bef-825d-d1ce99693438-tls-certs\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.753397 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.753337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l666w"] Aug 04 17:58:22.793183 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.793153 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pldx4"] Aug 04 17:58:22.796150 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:22.796125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fae65b2_3848_43fb_bf33_b9756d8e3d08.slice/crio-e1bef17eedab4dbe2dd48a4b04427f2d1486c1fd4a4b8300d2c0a303bda98bf0 WatchSource:0}: Error finding container e1bef17eedab4dbe2dd48a4b04427f2d1486c1fd4a4b8300d2c0a303bda98bf0: Status 404 returned error can't find the container with id e1bef17eedab4dbe2dd48a4b04427f2d1486c1fd4a4b8300d2c0a303bda98bf0 Aug 04 17:58:22.851446 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.851418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wwfm\" (UniqueName: \"kubernetes.io/projected/2ca8fc11-9ec3-4bef-825d-d1ce99693438-kube-api-access-9wwfm\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.851614 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.851453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8fc11-9ec3-4bef-825d-d1ce99693438-tls-certs\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.854409 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.854375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca8fc11-9ec3-4bef-825d-d1ce99693438-tls-certs\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:22.865579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:22.865549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wwfm\" (UniqueName: \"kubernetes.io/projected/2ca8fc11-9ec3-4bef-825d-d1ce99693438-kube-api-access-9wwfm\") pod \"model-serving-api-54d89bd5fc-l666w\" (UID: \"2ca8fc11-9ec3-4bef-825d-d1ce99693438\") " pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:23.050474 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.050433 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:23.189833 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.189764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" event={"ID":"9fae65b2-3848-43fb-bf33-b9756d8e3d08","Type":"ContainerStarted","Data":"e1bef17eedab4dbe2dd48a4b04427f2d1486c1fd4a4b8300d2c0a303bda98bf0"} Aug 04 17:58:23.218844 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.218800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-l666w"] Aug 04 17:58:23.220742 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:23.220708 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca8fc11_9ec3_4bef_825d_d1ce99693438.slice/crio-69af5bc41557d7999e895ac36a0f569144149955e5fa6ac4e937620a78f43b80 WatchSource:0}: Error finding container 69af5bc41557d7999e895ac36a0f569144149955e5fa6ac4e937620a78f43b80: Status 404 returned error can't find the container with id 69af5bc41557d7999e895ac36a0f569144149955e5fa6ac4e937620a78f43b80 Aug 04 17:58:23.418934 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.418851 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:23.423725 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.423698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.433000 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.432975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 17:58:23.433109 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.432980 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 17:58:23.433169 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.433102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 17:58:23.433169 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.433146 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 17:58:23.438078 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.438056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wqsxr\"" Aug 04 17:58:23.442926 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.442872 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxrr\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457350 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.457588 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.457453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.559525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.558044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.559525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.558114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.559525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.558153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.559525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.558197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.559525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.559343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.561378 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.560428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.561378 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.560485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxrr\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.561378 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.560538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.566727 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.566687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.567517 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.567414 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.568330 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.568310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.570833 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.570795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.573849 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.573781 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.575272 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.575235 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxrr\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr\") pod \"istiod-openshift-gateway-7b7995d5cd-jckn6\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:23.736768 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:23.736526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:24.197819 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:24.197781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" event={"ID":"2ca8fc11-9ec3-4bef-825d-d1ce99693438","Type":"ContainerStarted","Data":"69af5bc41557d7999e895ac36a0f569144149955e5fa6ac4e937620a78f43b80"} Aug 04 17:58:27.005300 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.005224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:27.007418 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:27.007388 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d3a4ea_a531_456c_aa58_67f43639f482.slice/crio-a62d1c6334ae7a4a3c702ef3b06a21fe1c784b6cc444ba39d60467e716bd326e WatchSource:0}: Error finding container a62d1c6334ae7a4a3c702ef3b06a21fe1c784b6cc444ba39d60467e716bd326e: Status 404 returned error can't find the container with id a62d1c6334ae7a4a3c702ef3b06a21fe1c784b6cc444ba39d60467e716bd326e Aug 04 17:58:27.216579 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.216451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" event={"ID":"59d3a4ea-a531-456c-aa58-67f43639f482","Type":"ContainerStarted","Data":"a62d1c6334ae7a4a3c702ef3b06a21fe1c784b6cc444ba39d60467e716bd326e"} Aug 04 17:58:27.218169 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.218141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" event={"ID":"79f2c87a-30c0-4652-a9a5-80baf5933464","Type":"ContainerStarted","Data":"9022d97059b404711c0400d8a23c05b4ee4e435d797db58d805132a36af84a3d"} Aug 04 17:58:27.219775 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.219749 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" event={"ID":"9fae65b2-3848-43fb-bf33-b9756d8e3d08","Type":"ContainerStarted","Data":"9a79b928de78629a1443cf16177ed9236017d8a55a0e831e853773aa7fced932"} Aug 04 17:58:27.219924 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.219898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:27.221447 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.221431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" event={"ID":"2ca8fc11-9ec3-4bef-825d-d1ce99693438","Type":"ContainerStarted","Data":"891523e2177f192638966cb460f52b39009a609de73360613026e36fda08143a"} Aug 04 17:58:27.221562 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.221535 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:27.240144 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.240098 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-798d66f497-jhv6z" podStartSLOduration=1.55678619 podStartE2EDuration="6.240083555s" totalImagesPullingTime="4.683297365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:27.237726448 +0000 UTC m=+368.979880508" watchObservedRunningTime="2026-08-04 17:58:27.240083555 +0000 UTC m=+368.982237613" Aug 04 17:58:27.258471 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.258430 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" podStartSLOduration=2.191024401 podStartE2EDuration="6.258414934s" totalImagesPullingTime="4.067390533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:27.258142956 +0000 UTC m=+369.000297014" watchObservedRunningTime="2026-08-04 17:58:27.258414934 +0000 UTC m=+369.000568992" Aug 04 17:58:27.285271 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.285222 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" podStartSLOduration=1.655581428 podStartE2EDuration="5.285206731s" totalImagesPullingTime="3.629625303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:27.284237044 +0000 UTC m=+369.026391102" watchObservedRunningTime="2026-08-04 17:58:27.285206731 +0000 UTC m=+369.027360788" Aug 04 17:58:27.505123 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.505050 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b94997cc-bzrqb"] Aug 04 17:58:27.508506 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.508477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.524631 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.524605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b94997cc-bzrqb"] Aug 04 17:58:27.601003 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.600970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-console-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601029 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-service-ca\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-oauth-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-trusted-ca-bundle\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnrx\" (UniqueName: \"kubernetes.io/projected/36be4884-2377-4d61-89e8-bc428dc49d01-kube-api-access-tbnrx\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.601349 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.601189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-oauth-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-console-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-service-ca\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-oauth-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-trusted-ca-bundle\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnrx\" (UniqueName: \"kubernetes.io/projected/36be4884-2377-4d61-89e8-bc428dc49d01-kube-api-access-tbnrx\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.702709 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.702705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-oauth-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.703323 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.703288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-console-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.703323 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.703306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-service-ca\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.703457 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.703378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-oauth-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.703457 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.703449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36be4884-2377-4d61-89e8-bc428dc49d01-trusted-ca-bundle\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.705085 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.705060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-oauth-config\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.705176 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.705142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36be4884-2377-4d61-89e8-bc428dc49d01-console-serving-cert\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.712215 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.712198 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnrx\" (UniqueName: \"kubernetes.io/projected/36be4884-2377-4d61-89e8-bc428dc49d01-kube-api-access-tbnrx\") pod \"console-64b94997cc-bzrqb\" (UID: \"36be4884-2377-4d61-89e8-bc428dc49d01\") " pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.817854 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.817817 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:27.952360 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:27.952341 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b94997cc-bzrqb"] Aug 04 17:58:27.954571 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:27.954532 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36be4884_2377_4d61_89e8_bc428dc49d01.slice/crio-f2d78c25ffb76bdeabbf9b8c2d54aeb53938cd84624cae40742db711bb44672f WatchSource:0}: Error finding container f2d78c25ffb76bdeabbf9b8c2d54aeb53938cd84624cae40742db711bb44672f: Status 404 returned error can't find the container with id f2d78c25ffb76bdeabbf9b8c2d54aeb53938cd84624cae40742db711bb44672f Aug 04 17:58:28.229631 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:28.229465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b94997cc-bzrqb" event={"ID":"36be4884-2377-4d61-89e8-bc428dc49d01","Type":"ContainerStarted","Data":"fc35735880e0ac779b4d234c6b9c5271decd5bf59ad1ae8a691e6853917eaeaf"} Aug 04 17:58:28.229631 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:28.229553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b94997cc-bzrqb" event={"ID":"36be4884-2377-4d61-89e8-bc428dc49d01","Type":"ContainerStarted","Data":"f2d78c25ffb76bdeabbf9b8c2d54aeb53938cd84624cae40742db711bb44672f"} Aug 04 17:58:28.254573 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:28.254521 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b94997cc-bzrqb" podStartSLOduration=1.254488808 podStartE2EDuration="1.254488808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:28.253778831 +0000 UTC m=+369.995932902" watchObservedRunningTime="2026-08-04 17:58:28.254488808 +0000 UTC m=+369.996642866" Aug 04 17:58:30.985677 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:30.985638 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 17:58:30.985996 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:30.985712 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 17:58:31.245617 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:31.245525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" event={"ID":"59d3a4ea-a531-456c-aa58-67f43639f482","Type":"ContainerStarted","Data":"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73"} Aug 04 17:58:31.245759 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:31.245639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:31.338230 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:31.338154 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" podStartSLOduration=4.362187364 podStartE2EDuration="8.338131678s" totalImagesPullingTime="3.975944314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:31.336902492 +0000 UTC m=+373.079056550" watchObservedRunningTime="2026-08-04 17:58:31.338131678 +0000 UTC m=+373.080285737" Aug 04 17:58:32.251878 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:32.251845 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-jckn6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 17:58:32.252297 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:32.251909 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 17:58:34.456703 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.456671 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7"] Aug 04 17:58:34.460265 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.460246 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.463797 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:58:34.463769 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="d8180508-a3f5-48d5-a440-788eef175f41" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.463797 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:58:34.463787 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"data-science-gateway-data-science-gateway-class-dockercfg-g5b46\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g5b46\"" type="*v1.Secret" Aug 04 17:58:34.499245 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.499216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7"] Aug 04 17:58:34.570490 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570655 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570655 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570613 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8180508-a3f5-48d5-a440-788eef175f41-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570655 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8180508-a3f5-48d5-a440-788eef175f41-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570655 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570799 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.570886 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.570868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszpx\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-kube-api-access-sszpx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671595 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671595 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8180508-a3f5-48d5-a440-788eef175f41-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8180508-a3f5-48d5-a440-788eef175f41-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.671832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672019 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672019 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.671973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszpx\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-kube-api-access-sszpx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672083 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672083 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672290 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672426 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672325 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672426 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.672752 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.672560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8180508-a3f5-48d5-a440-788eef175f41-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.674045 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.674027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8180508-a3f5-48d5-a440-788eef175f41-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.674369 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.674348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8180508-a3f5-48d5-a440-788eef175f41-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.689602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.689567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:34.689683 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:34.689654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszpx\" (UniqueName: \"kubernetes.io/projected/d8180508-a3f5-48d5-a440-788eef175f41-kube-api-access-sszpx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7\" (UID: \"d8180508-a3f5-48d5-a440-788eef175f41\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:35.251433 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:35.251406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:35.550729 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:35.550687 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g5b46\"" Aug 04 17:58:35.559920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:35.559891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:35.711586 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:35.711562 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7"] Aug 04 17:58:35.713657 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:35.713626 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8180508_a3f5_48d5_a440_788eef175f41.slice/crio-963a4c674f6ae547329e408a4f79f145cea15a959dcd07478f21d07331fbf030 WatchSource:0}: Error finding container 963a4c674f6ae547329e408a4f79f145cea15a959dcd07478f21d07331fbf030: Status 404 returned error can't find the container with id 963a4c674f6ae547329e408a4f79f145cea15a959dcd07478f21d07331fbf030 Aug 04 17:58:36.269599 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.269561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" event={"ID":"d8180508-a3f5-48d5-a440-788eef175f41","Type":"ContainerStarted","Data":"963a4c674f6ae547329e408a4f79f145cea15a959dcd07478f21d07331fbf030"} Aug 04 17:58:36.772849 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.772814 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8"] Aug 04 17:58:36.776382 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.776357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:36.802857 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.802832 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8"] Aug 04 17:58:36.807099 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.807075 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-frpn6\"" Aug 04 17:58:36.807233 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.807075 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 17:58:36.893551 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.893082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529511b-ad3f-408c-9344-b47586246793-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:36.893551 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.893165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhgd\" (UniqueName: \"kubernetes.io/projected/4529511b-ad3f-408c-9344-b47586246793-kube-api-access-vrhgd\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:36.994720 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.994686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhgd\" (UniqueName: \"kubernetes.io/projected/4529511b-ad3f-408c-9344-b47586246793-kube-api-access-vrhgd\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:36.994892 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.994782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529511b-ad3f-408c-9344-b47586246793-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:36.997696 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:36.997671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4529511b-ad3f-408c-9344-b47586246793-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:37.023371 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.023277 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhgd\" (UniqueName: \"kubernetes.io/projected/4529511b-ad3f-408c-9344-b47586246793-kube-api-access-vrhgd\") pod \"llmisvc-controller-manager-777c6bd7d5-4r8n8\" (UID: \"4529511b-ad3f-408c-9344-b47586246793\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:37.089020 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.088988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:58:37.271792 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.271630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8"] Aug 04 17:58:37.274838 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:58:37.274776 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4529511b_ad3f_408c_9344_b47586246793.slice/crio-46a8761611c60f23549cc4b864f47b749f46be9dfc295aa407fc0335109cb191 WatchSource:0}: Error finding container 46a8761611c60f23549cc4b864f47b749f46be9dfc295aa407fc0335109cb191: Status 404 returned error can't find the container with id 46a8761611c60f23549cc4b864f47b749f46be9dfc295aa407fc0335109cb191 Aug 04 17:58:37.641811 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.641780 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:37.642026 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.642000 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" containerID="cri-o://13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73" gracePeriod=30 Aug 04 17:58:37.818725 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.818691 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:37.819117 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.818752 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:37.823958 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:37.823936 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:38.232127 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.232097 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 17:58:38.234422 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.234399 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-l666w" Aug 04 17:58:38.250596 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.250556 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-jckn6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:8080/ready\": dial tcp 10.133.0.30:8080: connect: connection refused" start-of-body= Aug 04 17:58:38.250733 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.250629 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.30:8080/ready\": dial tcp 10.133.0.30:8080: connect: connection refused" Aug 04 17:58:38.282013 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.281974 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" event={"ID":"4529511b-ad3f-408c-9344-b47586246793","Type":"ContainerStarted","Data":"46a8761611c60f23549cc4b864f47b749f46be9dfc295aa407fc0335109cb191"} Aug 04 17:58:38.286535 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.286492 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b94997cc-bzrqb" Aug 04 17:58:38.506334 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.506304 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:58:38.536771 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.536729 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 17:58:38.536880 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.536861 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 17:58:38.536931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.536909 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 17:58:38.652370 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.652343 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:38.713435 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713380 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qxrr\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713435 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713546 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713621 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.713705 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.713653 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert\") pod \"59d3a4ea-a531-456c-aa58-67f43639f482\" (UID: \"59d3a4ea-a531-456c-aa58-67f43639f482\") " Aug 04 17:58:38.716088 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.715826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:58:38.718944 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.718914 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:58:38.719613 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.719567 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:58:38.719613 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.719568 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token" (OuterVolumeSpecName: "istio-token") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:58:38.719613 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.719610 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts" (OuterVolumeSpecName: "cacerts") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:58:38.719973 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.719943 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr" (OuterVolumeSpecName: "kube-api-access-6qxrr") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "kube-api-access-6qxrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:58:38.720151 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.720114 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs" (OuterVolumeSpecName: "local-certs") pod "59d3a4ea-a531-456c-aa58-67f43639f482" (UID: "59d3a4ea-a531-456c-aa58-67f43639f482"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:58:38.814564 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814535 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814564 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814565 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d3a4ea-a531-456c-aa58-67f43639f482-local-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814808 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814578 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-kubeconfig\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814808 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814585 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-cacerts\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814808 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814593 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-dns-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814808 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814600 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qxrr\" (UniqueName: \"kubernetes.io/projected/59d3a4ea-a531-456c-aa58-67f43639f482-kube-api-access-6qxrr\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:38.814808 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:38.814609 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/59d3a4ea-a531-456c-aa58-67f43639f482-istio-csr-ca-configmap\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:58:39.290257 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.289602 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" event={"ID":"d8180508-a3f5-48d5-a440-788eef175f41","Type":"ContainerStarted","Data":"039e97e83d997372076f8d829ae1ec061097eb7d29e49235139986680ed67fb9"} Aug 04 17:58:39.291136 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.291112 2559 generic.go:356] "Generic (PLEG): container finished" podID="59d3a4ea-a531-456c-aa58-67f43639f482" containerID="13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73" exitCode=0 Aug 04 17:58:39.291299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.291152 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" Aug 04 17:58:39.291299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.291198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" event={"ID":"59d3a4ea-a531-456c-aa58-67f43639f482","Type":"ContainerDied","Data":"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73"} Aug 04 17:58:39.291299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.291228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6" event={"ID":"59d3a4ea-a531-456c-aa58-67f43639f482","Type":"ContainerDied","Data":"a62d1c6334ae7a4a3c702ef3b06a21fe1c784b6cc444ba39d60467e716bd326e"} Aug 04 17:58:39.291299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.291247 2559 scope.go:117] "RemoveContainer" containerID="13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73" Aug 04 17:58:39.317455 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.317435 2559 scope.go:117] "RemoveContainer" containerID="13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73" Aug 04 17:58:39.317803 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:58:39.317778 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73\": container with ID starting with 13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73 not found: ID does not exist" containerID="13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73" Aug 04 17:58:39.317907 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.317816 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73"} err="failed to get container status \"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73\": rpc error: code = NotFound desc = could not find container \"13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73\": container with ID starting with 13118b385a2f7d254a7ea06ea2211e74940f0b6632a81cb04dbd1af9b7578a73 not found: ID does not exist" Aug 04 17:58:39.324996 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.324953 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" podStartSLOduration=2.504426044 podStartE2EDuration="5.324937826s" totalImagesPullingTime="2.820511782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:39.322018604 +0000 UTC m=+381.064172663" watchObservedRunningTime="2026-08-04 17:58:39.324937826 +0000 UTC m=+381.067091883" Aug 04 17:58:39.348036 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.348007 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:39.349737 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.349718 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-jckn6"] Aug 04 17:58:39.560778 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.560701 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:39.565866 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:39.565841 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:40.297299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:40.297266 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:40.298276 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:40.298260 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7" Aug 04 17:58:40.743708 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:40.743634 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" path="/var/lib/kubelet/pods/59d3a4ea-a531-456c-aa58-67f43639f482/volumes" Aug 04 17:58:41.303292 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:41.303255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" event={"ID":"4529511b-ad3f-408c-9344-b47586246793","Type":"ContainerStarted","Data":"82e727091b6b35499798747f47f52fff783b1fc924598f0b9838510069385c3c"} Aug 04 17:58:42.307670 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:58:42.307639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:59:03.526171 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.526119 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-784f988487-5x99j" podUID="7706572c-efce-4273-8186-7b508a399459" containerName="console" containerID="cri-o://55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5" gracePeriod=15 Aug 04 17:59:03.806338 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.806309 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:59:03.828313 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828117 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828313 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828187 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpfsz\" (UniqueName: \"kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828313 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828260 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828633 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828366 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828633 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828418 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828633 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828535 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.828791 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828752 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config" (OuterVolumeSpecName: "console-config") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:59:03.829176 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.828938 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-console-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.830044 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.830016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:59:03.830157 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.830062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca" (OuterVolumeSpecName: "service-ca") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:59:03.830802 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.830760 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" podStartSLOduration=23.9464685 podStartE2EDuration="27.830746487s" totalImagesPullingTime="3.884277987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:41.342749532 +0000 UTC m=+383.084903589" watchObservedRunningTime="2026-08-04 17:59:03.830746487 +0000 UTC m=+405.572900548" Aug 04 17:59:03.831587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.831560 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:59:03.831751 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.831728 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:59:03.832489 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.832469 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz" (OuterVolumeSpecName: "kube-api-access-hpfsz") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "kube-api-access-hpfsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:59:03.929891 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.929854 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert\") pod \"7706572c-efce-4273-8186-7b508a399459\" (UID: \"7706572c-efce-4273-8186-7b508a399459\") " Aug 04 17:59:03.930069 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930023 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-oauth-config\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.930069 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930034 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfsz\" (UniqueName: \"kubernetes.io/projected/7706572c-efce-4273-8186-7b508a399459-kube-api-access-hpfsz\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.930069 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930042 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-trusted-ca-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.930069 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930050 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.930069 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930058 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7706572c-efce-4273-8186-7b508a399459-console-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:03.930248 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:03.930186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7706572c-efce-4273-8186-7b508a399459" (UID: "7706572c-efce-4273-8186-7b508a399459"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:59:04.030748 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.030708 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7706572c-efce-4273-8186-7b508a399459-oauth-serving-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:04.415878 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.415842 2559 generic.go:356] "Generic (PLEG): container finished" podID="7706572c-efce-4273-8186-7b508a399459" containerID="55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5" exitCode=0 Aug 04 17:59:04.416076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.415892 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784f988487-5x99j" Aug 04 17:59:04.416076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.415922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784f988487-5x99j" event={"ID":"7706572c-efce-4273-8186-7b508a399459","Type":"ContainerDied","Data":"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5"} Aug 04 17:59:04.416076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.415969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784f988487-5x99j" event={"ID":"7706572c-efce-4273-8186-7b508a399459","Type":"ContainerDied","Data":"5aa20e30e29c01e2d63998d79a3ee0118e83f7805ac1ff586a6b66cef0acc822"} Aug 04 17:59:04.416076 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.415988 2559 scope.go:117] "RemoveContainer" containerID="55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5" Aug 04 17:59:04.424712 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.424696 2559 scope.go:117] "RemoveContainer" containerID="55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5" Aug 04 17:59:04.424976 ip-10-0-128-51 kubenswrapper[2559]: E0804 17:59:04.424953 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5\": container with ID starting with 55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5 not found: ID does not exist" containerID="55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5" Aug 04 17:59:04.425071 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.424980 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5"} err="failed to get container status \"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5\": rpc error: code = NotFound desc = could not find container \"55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5\": container with ID starting with 55a524d19c3043123939900a4574f037130eb0c465fdb20ed13f46f0b88ff6d5 not found: ID does not exist" Aug 04 17:59:04.438837 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.438813 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:59:04.444217 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.444189 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-784f988487-5x99j"] Aug 04 17:59:04.738842 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:04.738767 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7706572c-efce-4273-8186-7b508a399459" path="/var/lib/kubelet/pods/7706572c-efce-4273-8186-7b508a399459/volumes" Aug 04 17:59:13.314173 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:13.314140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 17:59:25.812541 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812511 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c"] Aug 04 17:59:25.812942 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812872 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7706572c-efce-4273-8186-7b508a399459" containerName="console" Aug 04 17:59:25.812942 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812889 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7706572c-efce-4273-8186-7b508a399459" containerName="console" Aug 04 17:59:25.812942 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812914 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" Aug 04 17:59:25.812942 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812920 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" Aug 04 17:59:25.813075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812970 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7706572c-efce-4273-8186-7b508a399459" containerName="console" Aug 04 17:59:25.813075 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.812980 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d3a4ea-a531-456c-aa58-67f43639f482" containerName="discovery" Aug 04 17:59:25.816162 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.816147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:25.819188 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.819157 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 17:59:25.819188 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.819173 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 17:59:25.819712 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.819695 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4l8z\"" Aug 04 17:59:25.826767 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.826745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c"] Aug 04 17:59:25.912437 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.912404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:25.912621 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.912454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdq5h\" (UniqueName: \"kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:25.912621 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.912485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:25.913220 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.913202 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x"] Aug 04 17:59:25.915672 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.915655 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:25.931602 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:25.931583 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x"] Aug 04 17:59:26.013800 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.013981 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdq5h\" (UniqueName: \"kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.013981 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.013981 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hrv\" (UniqueName: \"kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.013981 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.013981 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.013956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.014167 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.014156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.014202 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.014178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.026365 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.026331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdq5h\" (UniqueName: \"kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.030022 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.030002 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2"] Aug 04 17:59:26.032332 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.032311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.054686 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.054660 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2"] Aug 04 17:59:26.114867 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.114779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.114867 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.114838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.115141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.114910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hrv\" (UniqueName: \"kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.115141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.114981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.115141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.115051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssj4r\" (UniqueName: \"kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.115141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.115082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.115299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.115218 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.115299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.115273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.125221 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.125199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:26.128821 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.128799 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hrv\" (UniqueName: \"kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.171891 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.171854 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf"] Aug 04 17:59:26.175607 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.175583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.196551 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.196528 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf"] Aug 04 17:59:26.215606 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215576 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.215747 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.215816 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.215816 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw4p\" (UniqueName: \"kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.216141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssj4r\" (UniqueName: \"kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.216141 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.215854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.216363 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.216280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.217243 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.216661 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.224258 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.224237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:26.231596 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.231528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssj4r\" (UniqueName: \"kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.287668 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.287492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c"] Aug 04 17:59:26.290515 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:26.290463 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1280586_e1ea_4da1_8505_512f480b8b01.slice/crio-0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019 WatchSource:0}: Error finding container 0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019: Status 404 returned error can't find the container with id 0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019 Aug 04 17:59:26.317004 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.316978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.317116 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.317095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.317177 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.317147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw4p\" (UniqueName: \"kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.317569 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.317548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.317759 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.317736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.337112 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.337054 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw4p\" (UniqueName: \"kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.341881 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.341860 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:26.385776 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.385748 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x"] Aug 04 17:59:26.387714 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:26.387676 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d6904_53fc_4f2d_adb5_a2e2b4a4a40a.slice/crio-de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7 WatchSource:0}: Error finding container de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7: Status 404 returned error can't find the container with id de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7 Aug 04 17:59:26.489424 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.489393 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:26.490599 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.490581 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2"] Aug 04 17:59:26.499360 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:26.499335 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc167966f_cf8a_4ed7_b5ff_56ac95b60fcb.slice/crio-74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8 WatchSource:0}: Error finding container 74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8: Status 404 returned error can't find the container with id 74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8 Aug 04 17:59:26.518864 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.518811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" event={"ID":"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb","Type":"ContainerStarted","Data":"74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8"} Aug 04 17:59:26.520488 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.520461 2559 generic.go:356] "Generic (PLEG): container finished" podID="a1280586-e1ea-4da1-8505-512f480b8b01" containerID="0cc3380a70053b4a2966d02c702a9cfb5aa945269c84fde19d6c0552a6f3f0a0" exitCode=0 Aug 04 17:59:26.520608 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.520579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" event={"ID":"a1280586-e1ea-4da1-8505-512f480b8b01","Type":"ContainerDied","Data":"0cc3380a70053b4a2966d02c702a9cfb5aa945269c84fde19d6c0552a6f3f0a0"} Aug 04 17:59:26.520679 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.520623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" event={"ID":"a1280586-e1ea-4da1-8505-512f480b8b01","Type":"ContainerStarted","Data":"0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019"} Aug 04 17:59:26.522877 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.522855 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerStarted","Data":"cc8c20879c5546feff62f2e63673dc84733cd9de41498b3cc258181f649cdcfa"} Aug 04 17:59:26.522968 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.522894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerStarted","Data":"de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7"} Aug 04 17:59:26.644998 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:26.644935 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf"] Aug 04 17:59:26.688268 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:26.688239 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8442cdd7_4685_4bf2_a000_3784fd726cab.slice/crio-4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0 WatchSource:0}: Error finding container 4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0: Status 404 returned error can't find the container with id 4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0 Aug 04 17:59:27.529409 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.529378 2559 generic.go:356] "Generic (PLEG): container finished" podID="a1280586-e1ea-4da1-8505-512f480b8b01" containerID="303b514830f9f29c9e4ff9b1b1bdc7bfd7d970fbd7e4f1a7d9fd3eadfcf4cf80" exitCode=0 Aug 04 17:59:27.529804 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.529462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" event={"ID":"a1280586-e1ea-4da1-8505-512f480b8b01","Type":"ContainerDied","Data":"303b514830f9f29c9e4ff9b1b1bdc7bfd7d970fbd7e4f1a7d9fd3eadfcf4cf80"} Aug 04 17:59:27.531045 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.531023 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerID="cc8c20879c5546feff62f2e63673dc84733cd9de41498b3cc258181f649cdcfa" exitCode=0 Aug 04 17:59:27.531130 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.531107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerDied","Data":"cc8c20879c5546feff62f2e63673dc84733cd9de41498b3cc258181f649cdcfa"} Aug 04 17:59:27.532605 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.532584 2559 generic.go:356] "Generic (PLEG): container finished" podID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerID="61dc701201f8a37ad9ab09a307ee3116c398ece05079de3004d71af27298917f" exitCode=0 Aug 04 17:59:27.532692 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.532656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" event={"ID":"8442cdd7-4685-4bf2-a000-3784fd726cab","Type":"ContainerDied","Data":"61dc701201f8a37ad9ab09a307ee3116c398ece05079de3004d71af27298917f"} Aug 04 17:59:27.532744 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.532708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" event={"ID":"8442cdd7-4685-4bf2-a000-3784fd726cab","Type":"ContainerStarted","Data":"4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0"} Aug 04 17:59:27.534456 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.534438 2559 generic.go:356] "Generic (PLEG): container finished" podID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerID="964f21dd2dc3aa841c1509990d351218c1749de42fb437ad215653d6954fb067" exitCode=0 Aug 04 17:59:27.534564 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:27.534466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" event={"ID":"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb","Type":"ContainerDied","Data":"964f21dd2dc3aa841c1509990d351218c1749de42fb437ad215653d6954fb067"} Aug 04 17:59:28.541318 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.541277 2559 generic.go:356] "Generic (PLEG): container finished" podID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerID="0b2c51fd56649641ebf07e9ce7caf8a64a0c7f40309cd7c7ae5ed7ae332dedb3" exitCode=0 Aug 04 17:59:28.541713 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.541344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" event={"ID":"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb","Type":"ContainerDied","Data":"0b2c51fd56649641ebf07e9ce7caf8a64a0c7f40309cd7c7ae5ed7ae332dedb3"} Aug 04 17:59:28.546167 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.546065 2559 generic.go:356] "Generic (PLEG): container finished" podID="a1280586-e1ea-4da1-8505-512f480b8b01" containerID="d0c57daf7cc31df4a8afe838cc34063b45353c6e917299567c09036fb30ecfff" exitCode=0 Aug 04 17:59:28.546167 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.546157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" event={"ID":"a1280586-e1ea-4da1-8505-512f480b8b01","Type":"ContainerDied","Data":"d0c57daf7cc31df4a8afe838cc34063b45353c6e917299567c09036fb30ecfff"} Aug 04 17:59:28.548455 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.548435 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerID="f3dde6acf7b2980f2b1e713ce705c9a0d0c86bf9894ef6a2a2723b800bfdc367" exitCode=0 Aug 04 17:59:28.548587 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:28.548488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerDied","Data":"f3dde6acf7b2980f2b1e713ce705c9a0d0c86bf9894ef6a2a2723b800bfdc367"} Aug 04 17:59:29.555746 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.555713 2559 generic.go:356] "Generic (PLEG): container finished" podID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerID="e1ab8e266496e2595d6e21ebdc2cc2f961d1bb3d2a583b6ea88a97a367318b94" exitCode=0 Aug 04 17:59:29.556129 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.555805 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" event={"ID":"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb","Type":"ContainerDied","Data":"e1ab8e266496e2595d6e21ebdc2cc2f961d1bb3d2a583b6ea88a97a367318b94"} Aug 04 17:59:29.557939 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.557915 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerID="bf379a2f75c73fec3caf33cbb212f69618976f946ecd92eb07c3fc8b9b7873d9" exitCode=0 Aug 04 17:59:29.558049 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.557943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerDied","Data":"bf379a2f75c73fec3caf33cbb212f69618976f946ecd92eb07c3fc8b9b7873d9"} Aug 04 17:59:29.559644 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.559625 2559 generic.go:356] "Generic (PLEG): container finished" podID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerID="cb2b4a0fed598a9756de833334207aaaf0e0840c42e5f7ea838a511b0c4b1d98" exitCode=0 Aug 04 17:59:29.559754 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.559738 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" event={"ID":"8442cdd7-4685-4bf2-a000-3784fd726cab","Type":"ContainerDied","Data":"cb2b4a0fed598a9756de833334207aaaf0e0840c42e5f7ea838a511b0c4b1d98"} Aug 04 17:59:29.699483 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.699458 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:29.745756 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.745726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle\") pod \"a1280586-e1ea-4da1-8505-512f480b8b01\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " Aug 04 17:59:29.745909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.745801 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdq5h\" (UniqueName: \"kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h\") pod \"a1280586-e1ea-4da1-8505-512f480b8b01\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " Aug 04 17:59:29.745909 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.745890 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util\") pod \"a1280586-e1ea-4da1-8505-512f480b8b01\" (UID: \"a1280586-e1ea-4da1-8505-512f480b8b01\") " Aug 04 17:59:29.746223 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.746202 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle" (OuterVolumeSpecName: "bundle") pod "a1280586-e1ea-4da1-8505-512f480b8b01" (UID: "a1280586-e1ea-4da1-8505-512f480b8b01"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:29.747999 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.747977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h" (OuterVolumeSpecName: "kube-api-access-rdq5h") pod "a1280586-e1ea-4da1-8505-512f480b8b01" (UID: "a1280586-e1ea-4da1-8505-512f480b8b01"). InnerVolumeSpecName "kube-api-access-rdq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:59:29.750785 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.750764 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util" (OuterVolumeSpecName: "util") pod "a1280586-e1ea-4da1-8505-512f480b8b01" (UID: "a1280586-e1ea-4da1-8505-512f480b8b01"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:29.846638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.846553 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:29.846638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.846588 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdq5h\" (UniqueName: \"kubernetes.io/projected/a1280586-e1ea-4da1-8505-512f480b8b01-kube-api-access-rdq5h\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:29.846638 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:29.846602 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1280586-e1ea-4da1-8505-512f480b8b01-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.566262 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.566230 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" Aug 04 17:59:30.566262 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.566245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0wxh7c" event={"ID":"a1280586-e1ea-4da1-8505-512f480b8b01","Type":"ContainerDied","Data":"0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019"} Aug 04 17:59:30.566737 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.566292 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e4a5d76a3fff43cb8cf008687ceffcc41526d9f0474eb815cc257a477a77019" Aug 04 17:59:30.568445 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.568418 2559 generic.go:356] "Generic (PLEG): container finished" podID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerID="fa8c52231aaa182361404943ac3b0927b40c3376b7329299531217504abc2752" exitCode=0 Aug 04 17:59:30.568581 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.568517 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" event={"ID":"8442cdd7-4685-4bf2-a000-3784fd726cab","Type":"ContainerDied","Data":"fa8c52231aaa182361404943ac3b0927b40c3376b7329299531217504abc2752"} Aug 04 17:59:30.695662 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.695639 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:30.732248 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.732224 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:30.753637 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.753549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util\") pod \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " Aug 04 17:59:30.753790 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.753670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle\") pod \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " Aug 04 17:59:30.753790 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.753731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssj4r\" (UniqueName: \"kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r\") pod \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\" (UID: \"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb\") " Aug 04 17:59:30.754274 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.754247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle" (OuterVolumeSpecName: "bundle") pod "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" (UID: "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:30.756160 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.756130 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r" (OuterVolumeSpecName: "kube-api-access-ssj4r") pod "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" (UID: "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb"). InnerVolumeSpecName "kube-api-access-ssj4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:59:30.759258 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.759234 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util" (OuterVolumeSpecName: "util") pod "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" (UID: "c167966f-cf8a-4ed7-b5ff-56ac95b60fcb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:30.854682 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854607 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle\") pod \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " Aug 04 17:59:30.854682 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854651 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util\") pod \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " Aug 04 17:59:30.854682 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854673 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47hrv\" (UniqueName: \"kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv\") pod \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\" (UID: \"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a\") " Aug 04 17:59:30.854931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854819 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.854931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854830 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssj4r\" (UniqueName: \"kubernetes.io/projected/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-kube-api-access-ssj4r\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.854931 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.854840 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c167966f-cf8a-4ed7-b5ff-56ac95b60fcb-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.855193 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.855167 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle" (OuterVolumeSpecName: "bundle") pod "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" (UID: "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:30.856923 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.856898 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv" (OuterVolumeSpecName: "kube-api-access-47hrv") pod "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" (UID: "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a"). InnerVolumeSpecName "kube-api-access-47hrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:59:30.859377 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.859361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util" (OuterVolumeSpecName: "util") pod "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" (UID: "7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:30.955640 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.955613 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.955640 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.955634 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:30.955640 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:30.955643 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hrv\" (UniqueName: \"kubernetes.io/projected/7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a-kube-api-access-47hrv\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:31.575065 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.575039 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" Aug 04 17:59:31.575494 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.575076 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6mt2x" event={"ID":"7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a","Type":"ContainerDied","Data":"de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7"} Aug 04 17:59:31.575494 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.575125 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de253838747920a6c4c466cc3d6868b05b7d5983b3ac5e7edbe77a51b671afd7" Aug 04 17:59:31.577231 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.577207 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" event={"ID":"c167966f-cf8a-4ed7-b5ff-56ac95b60fcb","Type":"ContainerDied","Data":"74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8"} Aug 04 17:59:31.577231 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.577222 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870537hpl2" Aug 04 17:59:31.577352 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.577251 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fac9f6c42adc406a69a686a8420b9d858c5c3ecb13b83d0bd789174483b4b8" Aug 04 17:59:31.712132 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.712109 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:31.864292 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.864211 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle\") pod \"8442cdd7-4685-4bf2-a000-3784fd726cab\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " Aug 04 17:59:31.864292 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.864262 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util\") pod \"8442cdd7-4685-4bf2-a000-3784fd726cab\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " Aug 04 17:59:31.864470 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.864307 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhw4p\" (UniqueName: \"kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p\") pod \"8442cdd7-4685-4bf2-a000-3784fd726cab\" (UID: \"8442cdd7-4685-4bf2-a000-3784fd726cab\") " Aug 04 17:59:31.864862 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.864832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle" (OuterVolumeSpecName: "bundle") pod "8442cdd7-4685-4bf2-a000-3784fd726cab" (UID: "8442cdd7-4685-4bf2-a000-3784fd726cab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:31.866862 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.866836 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p" (OuterVolumeSpecName: "kube-api-access-bhw4p") pod "8442cdd7-4685-4bf2-a000-3784fd726cab" (UID: "8442cdd7-4685-4bf2-a000-3784fd726cab"). InnerVolumeSpecName "kube-api-access-bhw4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:59:31.870339 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.870311 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util" (OuterVolumeSpecName: "util") pod "8442cdd7-4685-4bf2-a000-3784fd726cab" (UID: "8442cdd7-4685-4bf2-a000-3784fd726cab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:59:31.965526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.965475 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-util\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:31.965526 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.965521 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw4p\" (UniqueName: \"kubernetes.io/projected/8442cdd7-4685-4bf2-a000-3784fd726cab-kube-api-access-bhw4p\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:31.965694 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:31.965536 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8442cdd7-4685-4bf2-a000-3784fd726cab-bundle\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 17:59:32.583279 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:32.583249 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" Aug 04 17:59:32.583279 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:32.583274 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4rshf" event={"ID":"8442cdd7-4685-4bf2-a000-3784fd726cab","Type":"ContainerDied","Data":"4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0"} Aug 04 17:59:32.583680 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:32.583318 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2d8bb9cf110fd00763835e14bb6a932762be0e66884c4d669ade1b989fbfe0" Aug 04 17:59:37.800084 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800043 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-lfqwt"] Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800379 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800389 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800398 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800403 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800412 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="util" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800418 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="util" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800423 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800428 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800436 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800441 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800449 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800453 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="extract" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800460 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800465 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="pull" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800470 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="util" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800475 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="util" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800480 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="util" Aug 04 17:59:37.800476 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800485 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="util" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800491 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="extract" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800510 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="extract" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800516 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="util" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800523 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="util" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800530 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="pull" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800537 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="pull" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800586 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c167966f-cf8a-4ed7-b5ff-56ac95b60fcb" containerName="extract" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800594 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5d6904-53fc-4f2d-adb5-a2e2b4a4a40a" containerName="extract" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800602 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1280586-e1ea-4da1-8505-512f480b8b01" containerName="extract" Aug 04 17:59:37.801061 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.800606 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8442cdd7-4685-4bf2-a000-3784fd726cab" containerName="extract" Aug 04 17:59:37.803719 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.803701 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:37.805920 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.805900 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 17:59:37.806032 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.805949 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 17:59:37.806032 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.805967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-65bzx\"" Aug 04 17:59:37.818516 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.818473 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-lfqwt"] Aug 04 17:59:37.919680 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:37.919650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfd9\" (UniqueName: \"kubernetes.io/projected/da150961-b2b1-4f17-a599-19507ad602ee-kube-api-access-bjfd9\") pod \"authorino-operator-68d88948d8-lfqwt\" (UID: \"da150961-b2b1-4f17-a599-19507ad602ee\") " pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:38.020465 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:38.020422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfd9\" (UniqueName: \"kubernetes.io/projected/da150961-b2b1-4f17-a599-19507ad602ee-kube-api-access-bjfd9\") pod \"authorino-operator-68d88948d8-lfqwt\" (UID: \"da150961-b2b1-4f17-a599-19507ad602ee\") " pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:38.037299 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:38.037267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfd9\" (UniqueName: \"kubernetes.io/projected/da150961-b2b1-4f17-a599-19507ad602ee-kube-api-access-bjfd9\") pod \"authorino-operator-68d88948d8-lfqwt\" (UID: \"da150961-b2b1-4f17-a599-19507ad602ee\") " pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:38.113538 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:38.113429 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:38.263618 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:38.263597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-lfqwt"] Aug 04 17:59:38.265411 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:38.265372 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda150961_b2b1_4f17_a599_19507ad602ee.slice/crio-494f96bd188c0af747f3fe0d53d7dee255e383f4dbeb3b76916492c3138d4865 WatchSource:0}: Error finding container 494f96bd188c0af747f3fe0d53d7dee255e383f4dbeb3b76916492c3138d4865: Status 404 returned error can't find the container with id 494f96bd188c0af747f3fe0d53d7dee255e383f4dbeb3b76916492c3138d4865 Aug 04 17:59:38.617525 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:38.617469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" event={"ID":"da150961-b2b1-4f17-a599-19507ad602ee","Type":"ContainerStarted","Data":"494f96bd188c0af747f3fe0d53d7dee255e383f4dbeb3b76916492c3138d4865"} Aug 04 17:59:41.635105 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:41.635064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" event={"ID":"da150961-b2b1-4f17-a599-19507ad602ee","Type":"ContainerStarted","Data":"6ac96efdbac2cead50f82f074148e4f07de2dd5349022ca1aaab6327716130db"} Aug 04 17:59:41.635544 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:41.635125 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:41.664463 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:41.664421 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" podStartSLOduration=2.088868945 podStartE2EDuration="4.664405684s" totalImagesPullingTime="2.575536739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:41.662316951 +0000 UTC m=+443.404471008" watchObservedRunningTime="2026-08-04 17:59:41.664405684 +0000 UTC m=+443.406559741" Aug 04 17:59:42.418280 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.415804 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w"] Aug 04 17:59:42.420767 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.420734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.436060 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.436042 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wg58v\"" Aug 04 17:59:42.447173 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.447141 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w"] Aug 04 17:59:42.562885 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.562853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzm9n\" (UniqueName: \"kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.563052 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.562927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.663463 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.663434 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzm9n\" (UniqueName: \"kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.663832 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.663493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.663916 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.663899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.688619 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.688567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzm9n\" (UniqueName: \"kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n\") pod \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.730124 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.730090 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:42.874151 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:42.874124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w"] Aug 04 17:59:42.875017 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:42.874992 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2304501d_d8ea_488f_b3bc_9c975a815fe8.slice/crio-0b0f16564b30976b762fc1f86f9ce1cd03119824abbb295de7bb12b3011d13a0 WatchSource:0}: Error finding container 0b0f16564b30976b762fc1f86f9ce1cd03119824abbb295de7bb12b3011d13a0: Status 404 returned error can't find the container with id 0b0f16564b30976b762fc1f86f9ce1cd03119824abbb295de7bb12b3011d13a0 Aug 04 17:59:43.646701 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:43.646663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" event={"ID":"2304501d-d8ea-488f-b3bc-9c975a815fe8","Type":"ContainerStarted","Data":"0b0f16564b30976b762fc1f86f9ce1cd03119824abbb295de7bb12b3011d13a0"} Aug 04 17:59:44.332895 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.332859 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-52rkg"] Aug 04 17:59:44.336869 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.336847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:44.339802 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.339780 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q74g9\"" Aug 04 17:59:44.340287 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.340267 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 17:59:44.347491 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.347472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-52rkg"] Aug 04 17:59:44.480990 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.480957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6zv\" (UniqueName: \"kubernetes.io/projected/9fe634df-a104-4a5e-b98f-b5118c091669-kube-api-access-gq6zv\") pod \"dns-operator-controller-manager-5985564776-52rkg\" (UID: \"9fe634df-a104-4a5e-b98f-b5118c091669\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:44.582367 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.582332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6zv\" (UniqueName: \"kubernetes.io/projected/9fe634df-a104-4a5e-b98f-b5118c091669-kube-api-access-gq6zv\") pod \"dns-operator-controller-manager-5985564776-52rkg\" (UID: \"9fe634df-a104-4a5e-b98f-b5118c091669\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:44.594904 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.594838 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6zv\" (UniqueName: \"kubernetes.io/projected/9fe634df-a104-4a5e-b98f-b5118c091669-kube-api-access-gq6zv\") pod \"dns-operator-controller-manager-5985564776-52rkg\" (UID: \"9fe634df-a104-4a5e-b98f-b5118c091669\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:44.649195 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.649168 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:44.807266 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:44.807239 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-52rkg"] Aug 04 17:59:44.809541 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:44.809488 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe634df_a104_4a5e_b98f_b5118c091669.slice/crio-f143356886d3134219afc3e58e5c1cb1d9e913a7add58cf33cd8336eeff19611 WatchSource:0}: Error finding container f143356886d3134219afc3e58e5c1cb1d9e913a7add58cf33cd8336eeff19611: Status 404 returned error can't find the container with id f143356886d3134219afc3e58e5c1cb1d9e913a7add58cf33cd8336eeff19611 Aug 04 17:59:45.661935 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:45.661895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" event={"ID":"9fe634df-a104-4a5e-b98f-b5118c091669","Type":"ContainerStarted","Data":"f143356886d3134219afc3e58e5c1cb1d9e913a7add58cf33cd8336eeff19611"} Aug 04 17:59:46.450918 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.450882 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6"] Aug 04 17:59:46.454899 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.454874 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:46.458333 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.458307 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gwzqr\"" Aug 04 17:59:46.468760 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.468736 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6"] Aug 04 17:59:46.604272 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.604227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vjh\" (UniqueName: \"kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh\") pod \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" (UID: \"632a37cf-f1b3-4b50-9781-5da3708d9452\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:46.705690 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.705610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vjh\" (UniqueName: \"kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh\") pod \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" (UID: \"632a37cf-f1b3-4b50-9781-5da3708d9452\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:46.720006 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.719968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vjh\" (UniqueName: \"kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh\") pod \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" (UID: \"632a37cf-f1b3-4b50-9781-5da3708d9452\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:46.770011 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.769984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:46.927296 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:46.927267 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6"] Aug 04 17:59:46.929754 ip-10-0-128-51 kubenswrapper[2559]: W0804 17:59:46.929726 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632a37cf_f1b3_4b50_9781_5da3708d9452.slice/crio-911722c4171d12c37f16b112082722337b2257d63d90b607aa14b8a5e6c8195e WatchSource:0}: Error finding container 911722c4171d12c37f16b112082722337b2257d63d90b607aa14b8a5e6c8195e: Status 404 returned error can't find the container with id 911722c4171d12c37f16b112082722337b2257d63d90b607aa14b8a5e6c8195e Aug 04 17:59:47.678597 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.678554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" event={"ID":"9fe634df-a104-4a5e-b98f-b5118c091669","Type":"ContainerStarted","Data":"8e509a628214a00a25fa816c9b1d5623e79b3f0b3c08114a774fdca0973bd989"} Aug 04 17:59:47.678754 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.678639 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:47.680458 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.680424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" event={"ID":"2304501d-d8ea-488f-b3bc-9c975a815fe8","Type":"ContainerStarted","Data":"31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a"} Aug 04 17:59:47.680597 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.680559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 17:59:47.681727 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.681705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" event={"ID":"632a37cf-f1b3-4b50-9781-5da3708d9452","Type":"ContainerStarted","Data":"911722c4171d12c37f16b112082722337b2257d63d90b607aa14b8a5e6c8195e"} Aug 04 17:59:47.705120 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.705078 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" podStartSLOduration=0.98639836 podStartE2EDuration="3.705062861s" totalImagesPullingTime="2.718664501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:47.700569648 +0000 UTC m=+449.442723697" watchObservedRunningTime="2026-08-04 17:59:47.705062861 +0000 UTC m=+449.447216919" Aug 04 17:59:47.735031 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:47.734986 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" podStartSLOduration=2.001785585 podStartE2EDuration="5.734967639s" totalImagesPullingTime="3.733182054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:47.731028235 +0000 UTC m=+449.473182293" watchObservedRunningTime="2026-08-04 17:59:47.734967639 +0000 UTC m=+449.477121699" Aug 04 17:59:49.489634 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.489604 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26"] Aug 04 17:59:49.492945 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.492924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.495441 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.495422 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Aug 04 17:59:49.495559 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.495477 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Aug 04 17:59:49.495664 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.495643 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g4hn2\"" Aug 04 17:59:49.500988 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.500961 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26"] Aug 04 17:59:49.529672 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.529643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz5n\" (UniqueName: \"kubernetes.io/projected/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-kube-api-access-tpz5n\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.529848 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.529708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.529848 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.529753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.631102 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.631071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.631284 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.631118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.631284 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.631205 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz5n\" (UniqueName: \"kubernetes.io/projected/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-kube-api-access-tpz5n\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.631812 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.631788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.633696 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.633678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.641388 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.641368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz5n\" (UniqueName: \"kubernetes.io/projected/4bcb83f6-63ff-443f-a0b1-8a0dbf15622d-kube-api-access-tpz5n\") pod \"kuadrant-console-plugin-598f86468b-lkr26\" (UID: \"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.803837 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.803802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" Aug 04 17:59:49.940557 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:49.937464 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26"] Aug 04 17:59:50.699064 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:50.699028 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" event={"ID":"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d","Type":"ContainerStarted","Data":"a67ac24923f3e4cfee2e49ff1162dc3ad8cd8f740c6ed08d062161bfe168d24b"} Aug 04 17:59:51.706977 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:51.706942 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" event={"ID":"632a37cf-f1b3-4b50-9781-5da3708d9452","Type":"ContainerStarted","Data":"2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44"} Aug 04 17:59:51.707663 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:51.707076 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 17:59:51.736768 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:51.736717 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" podStartSLOduration=2.006498215 podStartE2EDuration="5.736701019s" totalImagesPullingTime="3.730202804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:51.733800453 +0000 UTC m=+453.475954511" watchObservedRunningTime="2026-08-04 17:59:51.736701019 +0000 UTC m=+453.478855077" Aug 04 17:59:52.642786 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:52.642738 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 17:59:55.731478 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:55.731437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" event={"ID":"4bcb83f6-63ff-443f-a0b1-8a0dbf15622d","Type":"ContainerStarted","Data":"163388faecc0f74e6072043647629d38470741e7e1df7328663b018b0e1514ff"} Aug 04 17:59:55.750904 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:55.750857 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-lkr26" podStartSLOduration=1.407440071 podStartE2EDuration="6.750842817s" totalImagesPullingTime="5.343402746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:55.749197222 +0000 UTC m=+457.491351281" watchObservedRunningTime="2026-08-04 17:59:55.750842817 +0000 UTC m=+457.492996874" Aug 04 17:59:58.690250 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:58.690221 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-52rkg" Aug 04 17:59:58.690660 ip-10-0-128-51 kubenswrapper[2559]: I0804 17:59:58.690337 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.061212 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.061177 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w"] Aug 04 18:00:01.061614 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.061417 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" containerName="manager" containerID="cri-o://31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a" gracePeriod=2 Aug 04 18:00:01.072313 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.072283 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w"] Aug 04 18:00:01.088603 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.088581 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6"] Aug 04 18:00:01.088809 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.088782 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" containerName="manager" containerID="cri-o://2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44" gracePeriod=2 Aug 04 18:00:01.092621 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.092464 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.093916 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.093726 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.094159 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.094132 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.096066 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.095908 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:01.096411 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.096384 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" containerName="manager" Aug 04 18:00:01.096411 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.096403 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" containerName="manager" Aug 04 18:00:01.096618 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.096590 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" containerName="manager" Aug 04 18:00:01.098407 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.098385 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.100016 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.099994 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6"] Aug 04 18:00:01.100094 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.100084 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.101865 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.101816 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.104966 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.104942 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.106670 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.106638 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.114255 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.114233 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:01.122863 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.122840 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m"] Aug 04 18:00:01.123339 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.123325 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" containerName="manager" Aug 04 18:00:01.123409 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.123343 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" containerName="manager" Aug 04 18:00:01.123443 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.123422 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" containerName="manager" Aug 04 18:00:01.126808 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.126791 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.137441 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.137418 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.137623 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.137449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pph7p\" (UniqueName: \"kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.138853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.138768 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m"] Aug 04 18:00:01.143209 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.143175 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.145137 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.145106 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.238594 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.238555 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.238750 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.238663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pph7p\" (UniqueName: \"kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.238830 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.238814 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx2pz\" (UniqueName: \"kubernetes.io/projected/aef5e2ac-f46a-40f0-af21-3fd798a00236-kube-api-access-fx2pz\") pod \"limitador-operator-controller-manager-6c549b6d85-2bx7m\" (UID: \"aef5e2ac-f46a-40f0-af21-3fd798a00236\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.238956 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.238936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.253980 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.253938 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pph7p\" (UniqueName: \"kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p\") pod \"kuadrant-operator-controller-manager-58fc56757c-czhgd\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.324239 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.324167 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.327567 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.327537 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.330660 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.330634 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.339740 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.339721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx2pz\" (UniqueName: \"kubernetes.io/projected/aef5e2ac-f46a-40f0-af21-3fd798a00236-kube-api-access-fx2pz\") pod \"limitador-operator-controller-manager-6c549b6d85-2bx7m\" (UID: \"aef5e2ac-f46a-40f0-af21-3fd798a00236\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.340577 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.340561 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.342321 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.342302 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.343897 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.343879 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.351512 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.351477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx2pz\" (UniqueName: \"kubernetes.io/projected/aef5e2ac-f46a-40f0-af21-3fd798a00236-kube-api-access-fx2pz\") pod \"limitador-operator-controller-manager-6c549b6d85-2bx7m\" (UID: \"aef5e2ac-f46a-40f0-af21-3fd798a00236\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.440733 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.440706 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzm9n\" (UniqueName: \"kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n\") pod \"2304501d-d8ea-488f-b3bc-9c975a815fe8\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " Aug 04 18:00:01.440897 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.440757 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume\") pod \"2304501d-d8ea-488f-b3bc-9c975a815fe8\" (UID: \"2304501d-d8ea-488f-b3bc-9c975a815fe8\") " Aug 04 18:00:01.440897 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.440778 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4vjh\" (UniqueName: \"kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh\") pod \"632a37cf-f1b3-4b50-9781-5da3708d9452\" (UID: \"632a37cf-f1b3-4b50-9781-5da3708d9452\") " Aug 04 18:00:01.441218 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.441193 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2304501d-d8ea-488f-b3bc-9c975a815fe8" (UID: "2304501d-d8ea-488f-b3bc-9c975a815fe8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:00:01.443046 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.443024 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n" (OuterVolumeSpecName: "kube-api-access-xzm9n") pod "2304501d-d8ea-488f-b3bc-9c975a815fe8" (UID: "2304501d-d8ea-488f-b3bc-9c975a815fe8"). InnerVolumeSpecName "kube-api-access-xzm9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:00:01.443114 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.443043 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh" (OuterVolumeSpecName: "kube-api-access-d4vjh") pod "632a37cf-f1b3-4b50-9781-5da3708d9452" (UID: "632a37cf-f1b3-4b50-9781-5da3708d9452"). InnerVolumeSpecName "kube-api-access-d4vjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:00:01.506287 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.506252 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.512493 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.512471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.542215 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.542185 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzm9n\" (UniqueName: \"kubernetes.io/projected/2304501d-d8ea-488f-b3bc-9c975a815fe8-kube-api-access-xzm9n\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:00:01.542215 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.542213 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2304501d-d8ea-488f-b3bc-9c975a815fe8-extensions-socket-volume\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:00:01.542215 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.542223 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4vjh\" (UniqueName: \"kubernetes.io/projected/632a37cf-f1b3-4b50-9781-5da3708d9452-kube-api-access-d4vjh\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:00:01.668154 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.668133 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:01.670263 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:00:01.670238 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3689c4_722a_4ffb_9419_74c013fa7287.slice/crio-417a416c68bc4e02bac61c5e49949a0a66a998264f18bfe651cd903ba52dd7ec WatchSource:0}: Error finding container 417a416c68bc4e02bac61c5e49949a0a66a998264f18bfe651cd903ba52dd7ec: Status 404 returned error can't find the container with id 417a416c68bc4e02bac61c5e49949a0a66a998264f18bfe651cd903ba52dd7ec Aug 04 18:00:01.676432 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.676409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m"] Aug 04 18:00:01.678150 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:00:01.678120 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef5e2ac_f46a_40f0_af21_3fd798a00236.slice/crio-0f3845d905aac46d4bc5e62dba58c268f0b17e126e89743ffc2866f72a56ecb2 WatchSource:0}: Error finding container 0f3845d905aac46d4bc5e62dba58c268f0b17e126e89743ffc2866f72a56ecb2: Status 404 returned error can't find the container with id 0f3845d905aac46d4bc5e62dba58c268f0b17e126e89743ffc2866f72a56ecb2 Aug 04 18:00:01.762618 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.762582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" event={"ID":"8b3689c4-722a-4ffb-9419-74c013fa7287","Type":"ContainerStarted","Data":"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5"} Aug 04 18:00:01.762789 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.762637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" event={"ID":"8b3689c4-722a-4ffb-9419-74c013fa7287","Type":"ContainerStarted","Data":"417a416c68bc4e02bac61c5e49949a0a66a998264f18bfe651cd903ba52dd7ec"} Aug 04 18:00:01.762789 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.762688 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:01.764292 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.764270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" event={"ID":"aef5e2ac-f46a-40f0-af21-3fd798a00236","Type":"ContainerStarted","Data":"099dbbdfbe05f35763895d13775ee4e3c86c6a19e31b3b9777c491a354832db2"} Aug 04 18:00:01.764396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.764306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" event={"ID":"aef5e2ac-f46a-40f0-af21-3fd798a00236","Type":"ContainerStarted","Data":"0f3845d905aac46d4bc5e62dba58c268f0b17e126e89743ffc2866f72a56ecb2"} Aug 04 18:00:01.764396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.764361 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:01.765697 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.765675 2559 generic.go:356] "Generic (PLEG): container finished" podID="2304501d-d8ea-488f-b3bc-9c975a815fe8" containerID="31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a" exitCode=0 Aug 04 18:00:01.765769 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.765721 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.765834 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.765770 2559 scope.go:117] "RemoveContainer" containerID="31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a" Aug 04 18:00:01.767151 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.767132 2559 generic.go:356] "Generic (PLEG): container finished" podID="632a37cf-f1b3-4b50-9781-5da3708d9452" containerID="2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44" exitCode=0 Aug 04 18:00:01.767234 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.767164 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.775187 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.775172 2559 scope.go:117] "RemoveContainer" containerID="31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a" Aug 04 18:00:01.775451 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.775434 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a\": container with ID starting with 31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a not found: ID does not exist" containerID="31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a" Aug 04 18:00:01.775568 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.775464 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a"} err="failed to get container status \"31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a\": rpc error: code = NotFound desc = could not find container \"31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a\": container with ID starting with 31877e1c5b342b35559cc1c1f60f5fbca31062df6df80d77fe339838ccc3c59a not found: ID does not exist" Aug 04 18:00:01.775568 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.775488 2559 scope.go:117] "RemoveContainer" containerID="2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44" Aug 04 18:00:01.783716 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.783698 2559 scope.go:117] "RemoveContainer" containerID="2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44" Aug 04 18:00:01.784000 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.783975 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44\": container with ID starting with 2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44 not found: ID does not exist" containerID="2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44" Aug 04 18:00:01.784082 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.784006 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44"} err="failed to get container status \"2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44\": rpc error: code = NotFound desc = could not find container \"2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44\": container with ID starting with 2f3c5c2f9c2e9e112259d25908d9fa08eccf27e3978c5b83127fe41ff14a9a44 not found: ID does not exist" Aug 04 18:00:01.789112 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.789073 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" podStartSLOduration=0.789059899 podStartE2EDuration="789.059899ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:01.787638667 +0000 UTC m=+463.529792734" watchObservedRunningTime="2026-08-04 18:00:01.789059899 +0000 UTC m=+463.531213956" Aug 04 18:00:01.791547 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.791526 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.793841 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.793820 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:01.820642 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:01.820514 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" podStartSLOduration=0.820481914 podStartE2EDuration="820.481914ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:01.817452727 +0000 UTC m=+463.559606784" watchObservedRunningTime="2026-08-04 18:00:01.820481914 +0000 UTC m=+463.562635971" Aug 04 18:00:01.821402 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.821372 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:01.823118 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:01.823089 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:02.250684 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.250644 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:00:02.255610 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.255593 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.259884 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:02.259848 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-nrf4w\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-nrf4w" Aug 04 18:00:02.267896 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:02.267869 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-x9dr6\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-x9dr6" Aug 04 18:00:02.289677 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.289651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:00:02.351789 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.351696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.351789 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.351746 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh7t\" (UniqueName: \"kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.452884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.452843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.452884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.452884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh7t\" (UniqueName: \"kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.453258 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.453234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.469777 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.469748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh7t\" (UniqueName: \"kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t\") pod \"kuadrant-operator-controller-manager-68cf589589-g62bk\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.565029 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.564993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:02.720617 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.720582 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:00:02.736396 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:00:02.736363 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff77033_2a55_4b06_84bf_f1591b0d79fc.slice/crio-5293df7312d0097ab524cd0b56d7edc0d6d9656c683c27b78e593fb9e5707e7a WatchSource:0}: Error finding container 5293df7312d0097ab524cd0b56d7edc0d6d9656c683c27b78e593fb9e5707e7a: Status 404 returned error can't find the container with id 5293df7312d0097ab524cd0b56d7edc0d6d9656c683c27b78e593fb9e5707e7a Aug 04 18:00:02.739761 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.739734 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2304501d-d8ea-488f-b3bc-9c975a815fe8" path="/var/lib/kubelet/pods/2304501d-d8ea-488f-b3bc-9c975a815fe8/volumes" Aug 04 18:00:02.740124 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.740108 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="632a37cf-f1b3-4b50-9781-5da3708d9452" path="/var/lib/kubelet/pods/632a37cf-f1b3-4b50-9781-5da3708d9452/volumes" Aug 04 18:00:02.775881 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:02.775840 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" event={"ID":"5ff77033-2a55-4b06-84bf-f1591b0d79fc","Type":"ContainerStarted","Data":"5293df7312d0097ab524cd0b56d7edc0d6d9656c683c27b78e593fb9e5707e7a"} Aug 04 18:00:03.786898 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:03.786863 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" event={"ID":"5ff77033-2a55-4b06-84bf-f1591b0d79fc","Type":"ContainerStarted","Data":"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a"} Aug 04 18:00:03.787313 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:03.786964 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:03.820022 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:03.819967 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" podStartSLOduration=1.819949174 podStartE2EDuration="1.819949174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:03.818392595 +0000 UTC m=+465.560546652" watchObservedRunningTime="2026-08-04 18:00:03.819949174 +0000 UTC m=+465.562103287" Aug 04 18:00:12.778480 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:12.778452 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:00:12.778866 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:12.778538 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:14.794043 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:14.794010 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:00:14.859489 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:14.859448 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:14.859715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:14.859690 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" podUID="8b3689c4-722a-4ffb-9419-74c013fa7287" containerName="manager" containerID="cri-o://98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5" gracePeriod=10 Aug 04 18:00:15.113183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.113159 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:15.265970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.265938 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pph7p\" (UniqueName: \"kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p\") pod \"8b3689c4-722a-4ffb-9419-74c013fa7287\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " Aug 04 18:00:15.265970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.265984 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume\") pod \"8b3689c4-722a-4ffb-9419-74c013fa7287\" (UID: \"8b3689c4-722a-4ffb-9419-74c013fa7287\") " Aug 04 18:00:15.266393 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.266366 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8b3689c4-722a-4ffb-9419-74c013fa7287" (UID: "8b3689c4-722a-4ffb-9419-74c013fa7287"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:00:15.268077 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.268045 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p" (OuterVolumeSpecName: "kube-api-access-pph7p") pod "8b3689c4-722a-4ffb-9419-74c013fa7287" (UID: "8b3689c4-722a-4ffb-9419-74c013fa7287"). InnerVolumeSpecName "kube-api-access-pph7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:00:15.367309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.367235 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3689c4-722a-4ffb-9419-74c013fa7287-extensions-socket-volume\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:00:15.367309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.367258 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pph7p\" (UniqueName: \"kubernetes.io/projected/8b3689c4-722a-4ffb-9419-74c013fa7287-kube-api-access-pph7p\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:00:15.853351 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.853318 2559 generic.go:356] "Generic (PLEG): container finished" podID="8b3689c4-722a-4ffb-9419-74c013fa7287" containerID="98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5" exitCode=0 Aug 04 18:00:15.853837 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.853373 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" Aug 04 18:00:15.853837 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.853405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" event={"ID":"8b3689c4-722a-4ffb-9419-74c013fa7287","Type":"ContainerDied","Data":"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5"} Aug 04 18:00:15.853837 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.853457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd" event={"ID":"8b3689c4-722a-4ffb-9419-74c013fa7287","Type":"ContainerDied","Data":"417a416c68bc4e02bac61c5e49949a0a66a998264f18bfe651cd903ba52dd7ec"} Aug 04 18:00:15.853837 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.853481 2559 scope.go:117] "RemoveContainer" containerID="98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5" Aug 04 18:00:15.862547 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.862338 2559 scope.go:117] "RemoveContainer" containerID="98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5" Aug 04 18:00:15.862617 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:00:15.862595 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5\": container with ID starting with 98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5 not found: ID does not exist" containerID="98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5" Aug 04 18:00:15.862656 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.862618 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5"} err="failed to get container status \"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5\": rpc error: code = NotFound desc = could not find container \"98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5\": container with ID starting with 98810f7154f0f44cb4fa5635abc370e6279c97881d82c4b3bf634e31fc7456d5 not found: ID does not exist" Aug 04 18:00:15.879109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.879083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:15.884260 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:15.884237 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czhgd"] Aug 04 18:00:16.739166 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:16.739138 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3689c4-722a-4ffb-9419-74c013fa7287" path="/var/lib/kubelet/pods/8b3689c4-722a-4ffb-9419-74c013fa7287/volumes" Aug 04 18:00:31.339629 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.339590 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6"] Aug 04 18:00:31.340017 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.339941 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3689c4-722a-4ffb-9419-74c013fa7287" containerName="manager" Aug 04 18:00:31.340017 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.339951 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3689c4-722a-4ffb-9419-74c013fa7287" containerName="manager" Aug 04 18:00:31.340017 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.339999 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3689c4-722a-4ffb-9419-74c013fa7287" containerName="manager" Aug 04 18:00:31.360780 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.360752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.364998 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.364970 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lhqrm\"" Aug 04 18:00:31.367757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.367711 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6"] Aug 04 18:00:31.506144 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506328 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506168 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5c7z\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-kube-api-access-p5c7z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506328 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.506729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.506558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f2833-3643-484a-9fd9-95cf35f72f5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f2833-3643-484a-9fd9-95cf35f72f5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607431 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5c7z\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-kube-api-access-p5c7z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607579 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607953 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.607953 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.608046 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.607979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.608210 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.608187 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577f2833-3643-484a-9fd9-95cf35f72f5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.608327 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.608308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.609776 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.609752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.610020 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.610003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.623155 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.623130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5c7z\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-kube-api-access-p5c7z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.623411 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.623390 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577f2833-3643-484a-9fd9-95cf35f72f5f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-zq5d6\" (UID: \"577f2833-3643-484a-9fd9-95cf35f72f5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.672406 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.672372 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:31.811566 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.811535 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6"] Aug 04 18:00:31.812954 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:00:31.812931 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577f2833_3643_484a_9fd9_95cf35f72f5f.slice/crio-52fd3011b5c2e881c39d631f8a96951746253a2d5c91d07955b244e8e8979cf9 WatchSource:0}: Error finding container 52fd3011b5c2e881c39d631f8a96951746253a2d5c91d07955b244e8e8979cf9: Status 404 returned error can't find the container with id 52fd3011b5c2e881c39d631f8a96951746253a2d5c91d07955b244e8e8979cf9 Aug 04 18:00:31.815011 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.814982 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:00:31.815075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.815045 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:00:31.815075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.815071 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:00:31.946125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.946073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" event={"ID":"577f2833-3643-484a-9fd9-95cf35f72f5f","Type":"ContainerStarted","Data":"80240b7f182fef1479b29daa907039e34236bcd441f28b822548e6efc1fb8c14"} Aug 04 18:00:31.946307 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.946156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" event={"ID":"577f2833-3643-484a-9fd9-95cf35f72f5f","Type":"ContainerStarted","Data":"52fd3011b5c2e881c39d631f8a96951746253a2d5c91d07955b244e8e8979cf9"} Aug 04 18:00:31.982724 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:31.982668 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podStartSLOduration=0.982646822 podStartE2EDuration="982.646822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:31.982419966 +0000 UTC m=+493.724574024" watchObservedRunningTime="2026-08-04 18:00:31.982646822 +0000 UTC m=+493.724800882" Aug 04 18:00:32.672814 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:32.672781 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:32.677478 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:32.677458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:32.951291 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:32.951204 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:32.952221 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:32.952204 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:00:33.575183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.575151 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:00:33.578762 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.578742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.581433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.581412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 18:00:33.590179 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.590159 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:00:33.606890 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.606865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:00:33.730056 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.730021 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnd48\" (UniqueName: \"kubernetes.io/projected/44454a20-5faf-44f8-a208-8a486716e174-kube-api-access-hnd48\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.730442 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.730134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44454a20-5faf-44f8-a208-8a486716e174-config-file\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.831150 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.831059 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44454a20-5faf-44f8-a208-8a486716e174-config-file\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.831324 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.831165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnd48\" (UniqueName: \"kubernetes.io/projected/44454a20-5faf-44f8-a208-8a486716e174-kube-api-access-hnd48\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.831776 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.831759 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44454a20-5faf-44f8-a208-8a486716e174-config-file\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.841237 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.841208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnd48\" (UniqueName: \"kubernetes.io/projected/44454a20-5faf-44f8-a208-8a486716e174-kube-api-access-hnd48\") pod \"limitador-limitador-85d9db4d5d-m727d\" (UID: \"44454a20-5faf-44f8-a208-8a486716e174\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:33.890085 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:33.890051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:34.029757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:34.029731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:00:34.032413 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:00:34.032383 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44454a20_5faf_44f8_a208_8a486716e174.slice/crio-f7c3bf3d13ee841dd9d8a83320ac3b2ac9ceeb3c39c73880ea8ccd2904384aa7 WatchSource:0}: Error finding container f7c3bf3d13ee841dd9d8a83320ac3b2ac9ceeb3c39c73880ea8ccd2904384aa7: Status 404 returned error can't find the container with id f7c3bf3d13ee841dd9d8a83320ac3b2ac9ceeb3c39c73880ea8ccd2904384aa7 Aug 04 18:00:34.964671 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:34.964640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" event={"ID":"44454a20-5faf-44f8-a208-8a486716e174","Type":"ContainerStarted","Data":"f7c3bf3d13ee841dd9d8a83320ac3b2ac9ceeb3c39c73880ea8ccd2904384aa7"} Aug 04 18:00:35.971553 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:35.971493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" event={"ID":"44454a20-5faf-44f8-a208-8a486716e174","Type":"ContainerStarted","Data":"b4968de20872b6476653181948ee3c82dcefe4aa4aff48c6bbcacc24574ffba2"} Aug 04 18:00:35.971961 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:35.971648 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:00:35.992709 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:35.992662 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" podStartSLOduration=1.79866887 podStartE2EDuration="2.992647449s" totalImagesPullingTime="1.193978579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:35.990885194 +0000 UTC m=+497.733039247" watchObservedRunningTime="2026-08-04 18:00:35.992647449 +0000 UTC m=+497.734801535" Aug 04 18:00:46.978043 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:00:46.978014 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-m727d" Aug 04 18:01:47.900346 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:47.900311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:01:57.395974 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.395939 2559 generic.go:356] "Generic (PLEG): container finished" podID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerID="80240b7f182fef1479b29daa907039e34236bcd441f28b822548e6efc1fb8c14" exitCode=0 Aug 04 18:01:57.396426 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.396009 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" event={"ID":"577f2833-3643-484a-9fd9-95cf35f72f5f","Type":"ContainerDied","Data":"80240b7f182fef1479b29daa907039e34236bcd441f28b822548e6efc1fb8c14"} Aug 04 18:01:57.396426 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.396373 2559 scope.go:117] "RemoveContainer" containerID="80240b7f182fef1479b29daa907039e34236bcd441f28b822548e6efc1fb8c14" Aug 04 18:01:57.397344 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.397316 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:01:57.397434 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.397387 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:01:57.397434 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:57.397416 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:01:58.404129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:58.404097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" event={"ID":"577f2833-3643-484a-9fd9-95cf35f72f5f","Type":"ContainerStarted","Data":"ddb53aefb22b2d91070e66a0589d6a5ec37938b07a23459106e5cfe147b485fe"} Aug 04 18:01:58.672820 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:58.672714 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:01:58.673959 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:58.673929 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zq5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Aug 04 18:01:58.674097 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:58.673982 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podUID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Aug 04 18:01:59.672785 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:59.672749 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zq5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Aug 04 18:01:59.673206 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:01:59.672799 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podUID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Aug 04 18:02:00.672960 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:00.672926 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zq5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Aug 04 18:02:00.673342 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:00.672979 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podUID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Aug 04 18:02:01.673080 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:01.673040 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zq5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Aug 04 18:02:01.673480 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:01.673103 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podUID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Aug 04 18:02:02.672865 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:02.672825 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-zq5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Aug 04 18:02:02.673041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:02.672883 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" podUID="577f2833-3643-484a-9fd9-95cf35f72f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Aug 04 18:02:02.951634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:02.951532 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:02:03.677393 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:03.677367 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:02:03.678329 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:03.678309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-zq5d6" Aug 04 18:02:18.408079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:18.408046 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:02:32.768795 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:32.768756 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:02:45.517404 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:02:45.517321 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:03:01.000972 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.000941 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw"] Aug 04 18:03:01.003415 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.003400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.005882 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.005861 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 18:03:01.006339 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.006324 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 18:03:01.006458 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.006440 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dk8xl\"" Aug 04 18:03:01.006843 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.006790 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 18:03:01.022085 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.022062 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw"] Aug 04 18:03:01.120924 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.120883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.121100 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.121013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.121100 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.121053 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.121100 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.121094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.121323 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.121241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046dee85-87a1-4004-ae27-847eaa3e5269-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.121323 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.121307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pv4\" (UniqueName: \"kubernetes.io/projected/046dee85-87a1-4004-ae27-847eaa3e5269-kube-api-access-h7pv4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222223 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046dee85-87a1-4004-ae27-847eaa3e5269-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pv4\" (UniqueName: \"kubernetes.io/projected/046dee85-87a1-4004-ae27-847eaa3e5269-kube-api-access-h7pv4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222708 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222768 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.222822 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.222771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.224981 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.224945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046dee85-87a1-4004-ae27-847eaa3e5269-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.225143 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.225122 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046dee85-87a1-4004-ae27-847eaa3e5269-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.231490 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.231462 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pv4\" (UniqueName: \"kubernetes.io/projected/046dee85-87a1-4004-ae27-847eaa3e5269-kube-api-access-h7pv4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw\" (UID: \"046dee85-87a1-4004-ae27-847eaa3e5269\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.313633 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.313601 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:01.452716 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.452692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw"] Aug 04 18:03:01.454099 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:03:01.454072 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046dee85_87a1_4004_ae27_847eaa3e5269.slice/crio-fdafd03709e393425961a78d9b46839ea232ac307f0d77c3dbf6f6f42848b6ff WatchSource:0}: Error finding container fdafd03709e393425961a78d9b46839ea232ac307f0d77c3dbf6f6f42848b6ff: Status 404 returned error can't find the container with id fdafd03709e393425961a78d9b46839ea232ac307f0d77c3dbf6f6f42848b6ff Aug 04 18:03:01.455473 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.455456 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:03:01.757169 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:01.757071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" event={"ID":"046dee85-87a1-4004-ae27-847eaa3e5269","Type":"ContainerStarted","Data":"fdafd03709e393425961a78d9b46839ea232ac307f0d77c3dbf6f6f42848b6ff"} Aug 04 18:03:02.536201 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:02.536160 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:03:02.764436 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:02.764342 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" event={"ID":"046dee85-87a1-4004-ae27-847eaa3e5269","Type":"ContainerStarted","Data":"9bb3d0775c931ec71817c389292da4ca7fa68250746a485de164ec94b3f019ee"} Aug 04 18:03:02.797690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:02.797634 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" podStartSLOduration=1.84984847 podStartE2EDuration="2.797618453s" totalImagesPullingTime="947.769983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:03:02.796560484 +0000 UTC m=+644.538714521" watchObservedRunningTime="2026-08-04 18:03:02.797618453 +0000 UTC m=+644.539772504" Aug 04 18:03:11.188846 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.188814 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9"] Aug 04 18:03:11.191517 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.191468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.198195 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.198166 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 18:03:11.215427 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.215401 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9"] Aug 04 18:03:11.314167 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:11.314494 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314464 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:11.314742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.314788 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.314858 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.314900 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.314935 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.314979 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.314963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkhb\" (UniqueName: \"kubernetes.io/projected/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-kube-api-access-5vkhb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.329222 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.329188 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:11.419395 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkhb\" (UniqueName: \"kubernetes.io/projected/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-kube-api-access-5vkhb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419581 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419858 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419858 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419759 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.419858 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.419814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.420447 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.420423 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.422187 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.422167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.422294 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.422275 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.429014 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.428994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkhb\" (UniqueName: \"kubernetes.io/projected/805bc751-44a8-41f3-b9eb-f9b97a8a8dba-kube-api-access-5vkhb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nttk9\" (UID: \"805bc751-44a8-41f3-b9eb-f9b97a8a8dba\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.502908 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.502822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:11.643133 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.643107 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9"] Aug 04 18:03:11.645261 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:03:11.645233 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805bc751_44a8_41f3_b9eb_f9b97a8a8dba.slice/crio-912e376f62585443a552ae4332ae2ca3b45d3565f0452f7183d118c1bd33f1f1 WatchSource:0}: Error finding container 912e376f62585443a552ae4332ae2ca3b45d3565f0452f7183d118c1bd33f1f1: Status 404 returned error can't find the container with id 912e376f62585443a552ae4332ae2ca3b45d3565f0452f7183d118c1bd33f1f1 Aug 04 18:03:11.820899 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.820857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" event={"ID":"805bc751-44a8-41f3-b9eb-f9b97a8a8dba","Type":"ContainerStarted","Data":"912e376f62585443a552ae4332ae2ca3b45d3565f0452f7183d118c1bd33f1f1"} Aug 04 18:03:11.832387 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:11.832358 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw" Aug 04 18:03:12.827864 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:12.827827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" event={"ID":"805bc751-44a8-41f3-b9eb-f9b97a8a8dba","Type":"ContainerStarted","Data":"ccba43bb92acce6782a8eea7c5c8b0787305b52d4d267f42d151411f1c2211cf"} Aug 04 18:03:12.852059 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:12.852002 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" podStartSLOduration=1.6604831820000001 podStartE2EDuration="1.851986247s" totalImagesPullingTime="191.503065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:03:12.850543938 +0000 UTC m=+654.592697995" watchObservedRunningTime="2026-08-04 18:03:12.851986247 +0000 UTC m=+654.594140304" Aug 04 18:03:13.308358 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:13.308325 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:03:21.503615 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:21.503572 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:21.504049 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:21.503632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:21.515888 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:21.515853 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:21.887264 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:21.887230 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nttk9" Aug 04 18:03:26.038317 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:03:26.038286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:04:46.893272 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:04:46.893241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:04:55.606005 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:04:55.605969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:05:09.101524 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:05:09.101460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:06:47.213555 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:06:47.213517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:06:55.606405 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:06:55.606368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:07:57.397244 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:07:57.397208 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:04.495664 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:04.495630 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:19.796007 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:19.795970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:25.801844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:25.801807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:35.290739 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:35.290704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:42.498304 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:42.498218 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:08:55.823701 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:08:55.823664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:00.723720 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:00.723688 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:05.797844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:05.797805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:10.592946 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:10.592910 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:18.396230 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:18.396195 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:27.999469 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:27.999438 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:09:38.781360 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:09:38.781324 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:02.905470 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:02.905433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:13.390683 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:13.390588 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:19.993566 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:19.993527 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:28.662277 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:28.662238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:39.065748 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:39.065716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:44.490434 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:44.490391 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:10:52.585630 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:10:52.585595 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:04.093078 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:04.093041 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:11.122830 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:11.122791 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:22.521043 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:22.521001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:27.399633 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:27.399596 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:33.900285 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:33.900253 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:42.584021 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:42.583940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:47.620022 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:47.619985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:11:59.609942 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:11:59.609908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:12:53.905702 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:53.905666 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:12:53.906196 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:53.905903 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" podUID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" containerName="manager" containerID="cri-o://ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a" gracePeriod=10 Aug 04 18:12:54.156115 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.156056 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:12:54.242458 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.242432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnh7t\" (UniqueName: \"kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t\") pod \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " Aug 04 18:12:54.242628 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.242471 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume\") pod \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\" (UID: \"5ff77033-2a55-4b06-84bf-f1591b0d79fc\") " Aug 04 18:12:54.242891 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.242866 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ff77033-2a55-4b06-84bf-f1591b0d79fc" (UID: "5ff77033-2a55-4b06-84bf-f1591b0d79fc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:12:54.244567 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.244534 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t" (OuterVolumeSpecName: "kube-api-access-lnh7t") pod "5ff77033-2a55-4b06-84bf-f1591b0d79fc" (UID: "5ff77033-2a55-4b06-84bf-f1591b0d79fc"). InnerVolumeSpecName "kube-api-access-lnh7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:12:54.343309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.343274 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnh7t\" (UniqueName: \"kubernetes.io/projected/5ff77033-2a55-4b06-84bf-f1591b0d79fc-kube-api-access-lnh7t\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:12:54.343309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.343303 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ff77033-2a55-4b06-84bf-f1591b0d79fc-extensions-socket-volume\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:12:54.987798 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.987767 2559 generic.go:356] "Generic (PLEG): container finished" podID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" containerID="ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a" exitCode=0 Aug 04 18:12:54.988214 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.987811 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" Aug 04 18:12:54.988214 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.987862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" event={"ID":"5ff77033-2a55-4b06-84bf-f1591b0d79fc","Type":"ContainerDied","Data":"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a"} Aug 04 18:12:54.988214 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.987916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk" event={"ID":"5ff77033-2a55-4b06-84bf-f1591b0d79fc","Type":"ContainerDied","Data":"5293df7312d0097ab524cd0b56d7edc0d6d9656c683c27b78e593fb9e5707e7a"} Aug 04 18:12:54.988214 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.987943 2559 scope.go:117] "RemoveContainer" containerID="ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a" Aug 04 18:12:54.998070 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.998051 2559 scope.go:117] "RemoveContainer" containerID="ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a" Aug 04 18:12:54.998342 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:12:54.998324 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a\": container with ID starting with ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a not found: ID does not exist" containerID="ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a" Aug 04 18:12:54.998400 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:54.998351 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a"} err="failed to get container status \"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a\": rpc error: code = NotFound desc = could not find container \"ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a\": container with ID starting with ad71f3cfc6c0a804baa1c96fe42aef04c7b0236c39b509c1b7c179b0fe87ef5a not found: ID does not exist" Aug 04 18:12:55.007583 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:55.007553 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:12:55.014436 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:55.014416 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-g62bk"] Aug 04 18:12:56.739353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:12:56.739318 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" path="/var/lib/kubelet/pods/5ff77033-2a55-4b06-84bf-f1591b0d79fc/volumes" Aug 04 18:13:54.556192 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.556158 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:13:54.556690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.556579 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" containerName="manager" Aug 04 18:13:54.556690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.556591 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" containerName="manager" Aug 04 18:13:54.556690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.556648 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff77033-2a55-4b06-84bf-f1591b0d79fc" containerName="manager" Aug 04 18:13:54.559644 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.559629 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.562746 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.562725 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wg58v\"" Aug 04 18:13:54.576900 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.576874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:13:54.577273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.577256 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.577356 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.577342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7tl\" (UniqueName: \"kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.678305 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.678270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7tl\" (UniqueName: \"kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.678462 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.678324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.678726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.678708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.693355 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.693325 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7tl\" (UniqueName: \"kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl\") pod \"kuadrant-operator-controller-manager-68cf589589-mt8vz\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:54.869637 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:54.869551 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:55.011718 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:55.011690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:13:55.013312 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:13:55.013280 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb606e682_00c3_4df9_84a9_1ba2510e97c1.slice/crio-8328ce42ab4f6a326788d7cb1cbcc441e371d6037241e35481349cdfc3d73f4f WatchSource:0}: Error finding container 8328ce42ab4f6a326788d7cb1cbcc441e371d6037241e35481349cdfc3d73f4f: Status 404 returned error can't find the container with id 8328ce42ab4f6a326788d7cb1cbcc441e371d6037241e35481349cdfc3d73f4f Aug 04 18:13:55.310332 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:55.310295 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" event={"ID":"b606e682-00c3-4df9-84a9-1ba2510e97c1","Type":"ContainerStarted","Data":"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10"} Aug 04 18:13:55.310491 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:55.310374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" event={"ID":"b606e682-00c3-4df9-84a9-1ba2510e97c1","Type":"ContainerStarted","Data":"8328ce42ab4f6a326788d7cb1cbcc441e371d6037241e35481349cdfc3d73f4f"} Aug 04 18:13:55.310491 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:55.310396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:13:55.340206 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:13:55.340142 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" podStartSLOduration=1.340124201 podStartE2EDuration="1.340124201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:13:55.336250752 +0000 UTC m=+1297.078404810" watchObservedRunningTime="2026-08-04 18:13:55.340124201 +0000 UTC m=+1297.082278259" Aug 04 18:14:06.318881 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:14:06.318844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:14:14.191549 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:14:14.191487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:14:21.180241 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:14:21.180207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:14:43.183913 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:14:43.183828 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:20.510417 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:20.510380 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:28.802039 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:28.802006 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:34.998891 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:34.998852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:39.897472 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:39.897437 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:49.981288 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:49.981253 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:15:55.600529 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:15:55.600477 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:16:05.182782 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:16:05.182746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:16:10.290901 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:16:10.290805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:16:20.505715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:16:20.505678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:16:26.987820 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:16:26.987782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:16:36.305231 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:16:36.305191 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:17:01.140854 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:17:01.140814 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:17:06.304655 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:17:06.304618 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:17:18.594034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:17:18.594000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:17:28.193269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:17:28.193233 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:01.390795 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:01.390712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:10.986985 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:10.986952 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:21.882098 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:21.882056 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:31.687461 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:31.687425 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:36.989738 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:36.989701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:50.701783 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:50.701744 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:18:58.615915 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:18:58.615877 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:11.416276 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:11.416189 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:16.586793 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:16.586756 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:33.885833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:33.885795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:39.087295 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:39.087259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:52.386258 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:52.386221 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:19:57.490031 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:19:57.489997 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:04.391755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:04.391720 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:09.623155 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:09.623117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:18.294096 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:18.294065 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:26.988708 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:26.988668 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:33.904404 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:33.904368 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:43.989715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:43.989636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:20:50.982261 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:20:50.982220 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:01.199378 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:01.199337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:16.384987 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:16.384949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:23.324552 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:23.324522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:33.786116 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:33.786078 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:40.691017 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:40.690982 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:47.908044 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:47.908008 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:21:53.597037 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:21:53.597001 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:22:03.082166 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.082131 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:03.085561 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.085530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.088040 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.088017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-745b1b91-openshift-default-dockercfg-5whzn\"" Aug 04 18:22:03.110595 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.110569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:03.132906 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.132874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.132906 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.132910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.132963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.132989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8rg5\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.133018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.133084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133241 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.133120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133241 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.133197 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.133306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.133255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234413 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234413 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234413 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234432 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234559 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.234734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8rg5\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.235040 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234880 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.235040 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.234968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.235040 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.235009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.235157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.235071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.235418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.235401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.236910 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.236890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.237111 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.237095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.243432 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.243405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.243665 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.243645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8rg5\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5\") pod \"e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.395693 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.395591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:03.544688 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.544662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:03.546778 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:22:03.546751 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef88a228_7cdc_4545_b99c_45feab1c74c9.slice/crio-d86926867ec39d291ba48790f0e25f1cc764a0a3598d3a0cf6a90541aba42734 WatchSource:0}: Error finding container d86926867ec39d291ba48790f0e25f1cc764a0a3598d3a0cf6a90541aba42734: Status 404 returned error can't find the container with id d86926867ec39d291ba48790f0e25f1cc764a0a3598d3a0cf6a90541aba42734 Aug 04 18:22:03.548757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.548727 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:03.548839 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.548794 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:03.548839 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:03.548821 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:04.031244 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:04.031209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerStarted","Data":"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415"} Aug 04 18:22:04.031461 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:04.031269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerStarted","Data":"d86926867ec39d291ba48790f0e25f1cc764a0a3598d3a0cf6a90541aba42734"} Aug 04 18:22:04.053553 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:04.053476 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podStartSLOduration=1.053458431 podStartE2EDuration="1.053458431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:04.052094092 +0000 UTC m=+1785.794248150" watchObservedRunningTime="2026-08-04 18:22:04.053458431 +0000 UTC m=+1785.795612489" Aug 04 18:22:04.396494 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:04.396457 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:04.401755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:04.401730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:05.037165 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:05.037130 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:05.038028 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:05.038008 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:09.486542 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:09.486492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:22:19.114550 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.114487 2559 generic.go:356] "Generic (PLEG): container finished" podID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerID="2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415" exitCode=0 Aug 04 18:22:19.114963 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.114557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerDied","Data":"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415"} Aug 04 18:22:19.114963 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.114949 2559 scope.go:117] "RemoveContainer" containerID="2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415" Aug 04 18:22:19.115944 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.115920 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:19.116000 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.115988 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:19.116034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:19.116018 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:20.037102 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:20.037048 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:20.123823 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:20.123777 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerStarted","Data":"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac"} Aug 04 18:22:20.396646 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:20.396556 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:20.397427 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:20.397401 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:22:20.397581 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:20.397468 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:22:21.396887 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:21.396850 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:22:21.397284 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:21.396906 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:22:22.396688 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:22.396654 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:22:22.396860 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:22.396710 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:22:23.396620 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:23.396587 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:22:23.396991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:23.396638 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:22:24.396916 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:24.396877 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:22:24.397372 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:24.396941 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Aug 04 18:22:25.399754 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:25.399724 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:25.400210 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:25.400192 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:25.400751 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:25.400732 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:37.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:37.782442 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:37.782976 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:37.782715 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" containerID="cri-o://585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac" gracePeriod=30 Aug 04 18:22:38.774788 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.774751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:22:38.777337 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.777318 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.779538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.779522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-16a9c503-gw-openshift-default-dockercfg-gvn5c\"" Aug 04 18:22:38.792429 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.792406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:22:38.973262 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973262 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973544 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973544 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5s2f\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973544 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973544 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973559 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:38.973715 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:38.973616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.074926 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.074887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.074935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5s2f\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.074966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075399 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075399 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075399 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075583 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075583 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075583 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075556 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075707 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.075800 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.075782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.077389 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.077362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.077996 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.077974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.083929 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.083906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.084594 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.084491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5s2f\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f\") pod \"e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.087470 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.087452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:39.241245 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.241206 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:39.241557 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.241531 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:39.241709 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.241692 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:39.242524 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:39.242439 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:22:40.252288 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.252253 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerStarted","Data":"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e"} Aug 04 18:22:40.252747 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.252307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerStarted","Data":"ddf084cad2c3d5a5849765829f7bde6fc6567c33f425f29a0b1c848962032446"} Aug 04 18:22:40.276234 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.276179 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podStartSLOduration=2.2761577920000002 podStartE2EDuration="2.276157792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:40.27413557 +0000 UTC m=+1822.016289628" watchObservedRunningTime="2026-08-04 18:22:40.276157792 +0000 UTC m=+1822.018311852" Aug 04 18:22:40.291575 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.291547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:22:40.402189 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.402154 2559 patch_prober.go:28] interesting pod/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Aug 04 18:22:40.402363 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:40.402215 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:22:41.088057 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:41.088020 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:41.093297 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:41.093272 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:41.258441 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:41.258410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:41.259582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:41.259567 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:43.037353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.037327 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:43.109869 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109779 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.109869 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109824 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.109869 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109856 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109895 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109943 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.109966 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110018 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110043 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8rg5\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5\") pod \"ef88a228-7cdc-4545-b99c-45feab1c74c9\" (UID: \"ef88a228-7cdc-4545-b99c-45feab1c74c9\") " Aug 04 18:22:43.110420 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110196 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:22:43.110420 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110219 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:22:43.110420 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110333 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:22:43.110420 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110395 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:22:43.110634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110482 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:22:43.110634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110528 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef88a228-7cdc-4545-b99c-45feab1c74c9-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.110634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110546 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.110634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110557 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.110634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.110565 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.112205 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.112183 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:22:43.112352 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.112333 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5" (OuterVolumeSpecName: "kube-api-access-t8rg5") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "kube-api-access-t8rg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:22:43.112635 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.112614 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:22:43.112635 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.112614 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef88a228-7cdc-4545-b99c-45feab1c74c9" (UID: "ef88a228-7cdc-4545-b99c-45feab1c74c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:22:43.211229 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.211197 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.211229 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.211226 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.211229 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.211235 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.211455 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.211243 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef88a228-7cdc-4545-b99c-45feab1c74c9-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.211455 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.211252 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8rg5\" (UniqueName: \"kubernetes.io/projected/ef88a228-7cdc-4545-b99c-45feab1c74c9-kube-api-access-t8rg5\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:22:43.271740 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.271707 2559 generic.go:356] "Generic (PLEG): container finished" podID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerID="585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac" exitCode=0 Aug 04 18:22:43.271905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.271758 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" Aug 04 18:22:43.271905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.271792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerDied","Data":"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac"} Aug 04 18:22:43.271905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.271856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb" event={"ID":"ef88a228-7cdc-4545-b99c-45feab1c74c9","Type":"ContainerDied","Data":"d86926867ec39d291ba48790f0e25f1cc764a0a3598d3a0cf6a90541aba42734"} Aug 04 18:22:43.271905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.271880 2559 scope.go:117] "RemoveContainer" containerID="585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac" Aug 04 18:22:43.281746 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.281731 2559 scope.go:117] "RemoveContainer" containerID="2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415" Aug 04 18:22:43.289800 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.289781 2559 scope.go:117] "RemoveContainer" containerID="585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac" Aug 04 18:22:43.290080 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:22:43.290060 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac\": container with ID starting with 585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac not found: ID does not exist" containerID="585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac" Aug 04 18:22:43.290143 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.290089 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac"} err="failed to get container status \"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac\": rpc error: code = NotFound desc = could not find container \"585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac\": container with ID starting with 585f4390d578f93bfdd44eb2928687ec56efccb87ddbb950970d3a36de8a43ac not found: ID does not exist" Aug 04 18:22:43.290143 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.290105 2559 scope.go:117] "RemoveContainer" containerID="2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415" Aug 04 18:22:43.290357 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:22:43.290336 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415\": container with ID starting with 2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415 not found: ID does not exist" containerID="2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415" Aug 04 18:22:43.290424 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.290366 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415"} err="failed to get container status \"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415\": rpc error: code = NotFound desc = could not find container \"2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415\": container with ID starting with 2239220fd21a203db4646b06d5739e34b1325e4df204fd04b8d145ba3b63b415 not found: ID does not exist" Aug 04 18:22:43.295792 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.295768 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:43.300635 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:43.300614 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-745b1b91-openshift-default-54cd68b646-w24kb"] Aug 04 18:22:44.739540 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:44.739483 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" path="/var/lib/kubelet/pods/ef88a228-7cdc-4545-b99c-45feab1c74c9/volumes" Aug 04 18:22:49.927753 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:49.927709 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:22:54.336267 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.336231 2559 generic.go:356] "Generic (PLEG): container finished" podID="934c40a8-9941-46f7-970d-d2cddb89e454" containerID="0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e" exitCode=0 Aug 04 18:22:54.336799 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.336310 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerDied","Data":"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e"} Aug 04 18:22:54.336933 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.336915 2559 scope.go:117] "RemoveContainer" containerID="0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e" Aug 04 18:22:54.337940 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.337913 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:54.338041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.337995 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:54.338104 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:54.338041 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:22:55.347426 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:55.347386 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerStarted","Data":"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18"} Aug 04 18:22:56.088330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:56.088295 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:56.089509 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:56.089467 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:22:56.089630 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:56.089546 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:22:56.258951 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:56.258915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:22:57.087996 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:57.087957 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:22:57.088400 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:57.088015 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:22:58.088879 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:58.088838 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:22:58.089293 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:58.088926 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:22:59.088705 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:59.088668 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:22:59.088883 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:22:59.088731 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:23:00.088525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:00.088456 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Aug 04 18:23:00.088908 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:00.088538 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Aug 04 18:23:01.091895 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:01.091866 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:23:01.092965 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:01.092946 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:23:13.131111 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.131071 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:23:13.131901 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.131867 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" containerID="cri-o://519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18" gracePeriod=30 Aug 04 18:23:13.872274 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.872242 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:13.872792 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.872771 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:13.872792 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.872793 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:13.872965 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.872911 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:13.872965 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.872924 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:13.875415 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.875395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:13.877844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.877824 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bcb508fd-openshift-default-dockercfg-k62wd\"" Aug 04 18:23:13.887816 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:13.887779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:14.021803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.021768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.021803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.021807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hvz\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.021828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.021896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.022013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022243 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.022074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022243 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.022173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022243 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.022229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.022396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.022265 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123074 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.122972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123074 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123074 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hvz\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123353 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123244 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123698 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123698 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.123966 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.124135 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.123971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.124406 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.124382 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.126377 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.126354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.126377 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.126377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.135360 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.135333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.135704 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.135598 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hvz\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz\") pod \"e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.186340 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.186306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:14.330976 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.330939 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:14.332304 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:23:14.332245 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f1540c_dc98_4c8c_8fd6_7a1233d442f3.slice/crio-3db3f3e612f636303d7123fb8e7fde5ab411f3dd319a40ae349f43adc3d3607a WatchSource:0}: Error finding container 3db3f3e612f636303d7123fb8e7fde5ab411f3dd319a40ae349f43adc3d3607a: Status 404 returned error can't find the container with id 3db3f3e612f636303d7123fb8e7fde5ab411f3dd319a40ae349f43adc3d3607a Aug 04 18:23:14.335227 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.335188 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:14.335315 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.335271 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:14.335372 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.335312 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:14.453179 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.453144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerStarted","Data":"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7"} Aug 04 18:23:14.453382 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.453206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerStarted","Data":"3db3f3e612f636303d7123fb8e7fde5ab411f3dd319a40ae349f43adc3d3607a"} Aug 04 18:23:14.476992 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:14.476939 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podStartSLOduration=1.476921662 podStartE2EDuration="1.476921662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:14.474955434 +0000 UTC m=+1856.217109494" watchObservedRunningTime="2026-08-04 18:23:14.476921662 +0000 UTC m=+1856.219075721" Aug 04 18:23:15.186676 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:15.186640 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:15.191266 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:15.191244 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:15.460153 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:15.460063 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:15.461102 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:15.461083 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:15.592555 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:15.592519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:23:16.094780 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:16.094746 2559 patch_prober.go:28] interesting pod/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:23:16.094940 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:16.094814 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:23:18.385226 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.385200 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:23:18.462224 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5s2f\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462224 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462194 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462273 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462307 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462330 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462386 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462433 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462450 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data\") pod \"934c40a8-9941-46f7-970d-d2cddb89e454\" (UID: \"934c40a8-9941-46f7-970d-d2cddb89e454\") " Aug 04 18:23:18.462803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462613 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:18.462803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462753 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:18.462803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462753 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:23:18.462803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462792 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data" (OuterVolumeSpecName: "istio-data") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:18.463213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462926 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.463213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462948 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/934c40a8-9941-46f7-970d-d2cddb89e454-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.463213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462963 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.463213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.462978 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.463213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.463056 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:18.464586 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.464554 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token" (OuterVolumeSpecName: "istio-token") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:23:18.464711 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.464668 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:18.464760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.464720 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f" (OuterVolumeSpecName: "kube-api-access-j5s2f") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "kube-api-access-j5s2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:23:18.464846 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.464827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "934c40a8-9941-46f7-970d-d2cddb89e454" (UID: "934c40a8-9941-46f7-970d-d2cddb89e454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:23:18.479229 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.479199 2559 generic.go:356] "Generic (PLEG): container finished" podID="934c40a8-9941-46f7-970d-d2cddb89e454" containerID="519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18" exitCode=0 Aug 04 18:23:18.479333 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.479254 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" Aug 04 18:23:18.479333 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.479281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerDied","Data":"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18"} Aug 04 18:23:18.479416 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.479340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8" event={"ID":"934c40a8-9941-46f7-970d-d2cddb89e454","Type":"ContainerDied","Data":"ddf084cad2c3d5a5849765829f7bde6fc6567c33f425f29a0b1c848962032446"} Aug 04 18:23:18.479416 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.479365 2559 scope.go:117] "RemoveContainer" containerID="519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18" Aug 04 18:23:18.489070 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.489054 2559 scope.go:117] "RemoveContainer" containerID="0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e" Aug 04 18:23:18.497634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.497618 2559 scope.go:117] "RemoveContainer" containerID="519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18" Aug 04 18:23:18.497895 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:23:18.497876 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18\": container with ID starting with 519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18 not found: ID does not exist" containerID="519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18" Aug 04 18:23:18.497946 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.497905 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18"} err="failed to get container status \"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18\": rpc error: code = NotFound desc = could not find container \"519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18\": container with ID starting with 519d5ad375d0242d5f2db403fd64113331d9ad75b9cd020e63faf787c23eed18 not found: ID does not exist" Aug 04 18:23:18.497946 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.497923 2559 scope.go:117] "RemoveContainer" containerID="0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e" Aug 04 18:23:18.498128 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:23:18.498109 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e\": container with ID starting with 0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e not found: ID does not exist" containerID="0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e" Aug 04 18:23:18.498183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.498138 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e"} err="failed to get container status \"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e\": rpc error: code = NotFound desc = could not find container \"0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e\": container with ID starting with 0768b76c18bfecdfe75dc9011193aacea5189e5fe34d3ee270b9e42a9aedf53e not found: ID does not exist" Aug 04 18:23:18.502385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.502364 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:23:18.507034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.507010 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-16a9c503-gw-openshift-default-c4765cb56-6wnz8"] Aug 04 18:23:18.563386 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.563353 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/934c40a8-9941-46f7-970d-d2cddb89e454-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.563386 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.563385 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5s2f\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-kube-api-access-j5s2f\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.563634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.563395 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/934c40a8-9941-46f7-970d-d2cddb89e454-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.563634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.563406 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.563634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.563414 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/934c40a8-9941-46f7-970d-d2cddb89e454-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:18.748441 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:18.748363 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" path="/var/lib/kubelet/pods/934c40a8-9941-46f7-970d-d2cddb89e454/volumes" Aug 04 18:23:25.408467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:25.408427 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:23:29.542061 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.542026 2559 generic.go:356] "Generic (PLEG): container finished" podID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerID="0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7" exitCode=0 Aug 04 18:23:29.542616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.542126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerDied","Data":"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7"} Aug 04 18:23:29.542686 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.542625 2559 scope.go:117] "RemoveContainer" containerID="0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7" Aug 04 18:23:29.543783 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.543751 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:29.543908 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.543840 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:29.543908 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:29.543890 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:30.460617 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:30.460577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:30.551578 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:30.551547 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerStarted","Data":"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4"} Aug 04 18:23:31.187311 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:31.187275 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:31.188367 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:31.188344 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:23:31.188481 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:31.188396 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:23:32.187356 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:32.187316 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:23:32.187770 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:32.187378 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:23:33.187193 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:33.187152 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:23:33.187368 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:33.187221 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:23:34.187723 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:34.187688 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:23:34.188138 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:34.187756 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:23:35.186822 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:35.186782 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Aug 04 18:23:35.186987 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:35.186855 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Aug 04 18:23:36.191493 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:36.191461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:36.191915 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:36.191752 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:36.192418 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:36.192399 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:49.223530 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.223407 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:49.224061 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.223725 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" containerID="cri-o://3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4" gracePeriod=30 Aug 04 18:23:49.951586 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.951547 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:23:49.952000 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.951987 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.952048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952002 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.952048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952021 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.952048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952027 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.952048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952035 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:49.952048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952041 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef88a228-7cdc-4545-b99c-45feab1c74c9" containerName="istio-proxy" Aug 04 18:23:49.952201 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952116 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.952201 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.952122 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="934c40a8-9941-46f7-970d-d2cddb89e454" containerName="istio-proxy" Aug 04 18:23:49.954325 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.954300 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:49.956629 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.956607 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-07becd73-openshift-default-dockercfg-c9mhw\"" Aug 04 18:23:49.968553 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:49.968523 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:23:50.076513 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076736 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076527 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076736 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q5d\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076736 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076887 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076887 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076826 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076957 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.076957 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.077022 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.076955 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.177737 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.177737 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q5d\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.177901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178763 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178763 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178763 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.178763 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.178800 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.180719 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.180690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.181066 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.181046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.190953 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.190917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.191139 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.191112 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q5d\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d\") pod \"e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.265986 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.265885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:50.425130 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.425099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:23:50.427513 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:23:50.427450 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8eedb4_c456_46a3_90ab_746bc726270a.slice/crio-0e42d2c8772794a665ddaa183e3e22da47c135f901f5584ebad5dc11887409ee WatchSource:0}: Error finding container 0e42d2c8772794a665ddaa183e3e22da47c135f901f5584ebad5dc11887409ee: Status 404 returned error can't find the container with id 0e42d2c8772794a665ddaa183e3e22da47c135f901f5584ebad5dc11887409ee Aug 04 18:23:50.429994 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.429955 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:50.430080 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.430036 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:50.430080 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.430065 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:23:50.671669 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.671625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerStarted","Data":"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6"} Aug 04 18:23:50.672089 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.671685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerStarted","Data":"0e42d2c8772794a665ddaa183e3e22da47c135f901f5584ebad5dc11887409ee"} Aug 04 18:23:50.721825 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:50.721742 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podStartSLOduration=1.721717285 podStartE2EDuration="1.721717285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:50.718731824 +0000 UTC m=+1892.460885884" watchObservedRunningTime="2026-08-04 18:23:50.721717285 +0000 UTC m=+1892.463871349" Aug 04 18:23:51.195604 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.195560 2559 patch_prober.go:28] interesting pod/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:23:51.195792 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.195624 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:23:51.196308 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.196286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:23:51.266844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.266795 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:51.272186 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.272155 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:51.677563 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.677526 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:51.678491 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:51.678473 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:23:54.487701 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.487625 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:54.624351 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624309 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624452 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624488 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624568 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6hvz\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624631 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624669 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624727 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624758 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo\") pod \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\" (UID: \"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3\") " Aug 04 18:23:54.624853 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624801 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:54.625171 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.624862 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:23:54.625171 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.625118 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:54.625296 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.625183 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.625296 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.625203 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.625296 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.625274 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:54.625444 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.625415 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:54.627330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.627293 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:23:54.627586 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.627560 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:23:54.627586 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.627570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:23:54.627944 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.627924 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz" (OuterVolumeSpecName: "kube-api-access-l6hvz") pod "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" (UID: "c0f1540c-dc98-4c8c-8fd6-7a1233d442f3"). InnerVolumeSpecName "kube-api-access-l6hvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:23:54.696398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.696358 2559 generic.go:356] "Generic (PLEG): container finished" podID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerID="3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4" exitCode=0 Aug 04 18:23:54.696639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.696426 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" Aug 04 18:23:54.696639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.696438 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerDied","Data":"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4"} Aug 04 18:23:54.696639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.696527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh" event={"ID":"c0f1540c-dc98-4c8c-8fd6-7a1233d442f3","Type":"ContainerDied","Data":"3db3f3e612f636303d7123fb8e7fde5ab411f3dd319a40ae349f43adc3d3607a"} Aug 04 18:23:54.696639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.696574 2559 scope.go:117] "RemoveContainer" containerID="3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4" Aug 04 18:23:54.707372 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.707350 2559 scope.go:117] "RemoveContainer" containerID="0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7" Aug 04 18:23:54.716729 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.716701 2559 scope.go:117] "RemoveContainer" containerID="3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4" Aug 04 18:23:54.717992 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:23:54.717961 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4\": container with ID starting with 3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4 not found: ID does not exist" containerID="3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4" Aug 04 18:23:54.718105 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.718004 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4"} err="failed to get container status \"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4\": rpc error: code = NotFound desc = could not find container \"3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4\": container with ID starting with 3cf62fd81526596626d47db9019c53e638997317cd513d27d4c9d07e0b977bc4 not found: ID does not exist" Aug 04 18:23:54.718105 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.718056 2559 scope.go:117] "RemoveContainer" containerID="0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7" Aug 04 18:23:54.718324 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:23:54.718300 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7\": container with ID starting with 0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7 not found: ID does not exist" containerID="0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7" Aug 04 18:23:54.718387 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.718335 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7"} err="failed to get container status \"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7\": rpc error: code = NotFound desc = could not find container \"0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7\": container with ID starting with 0b2952ef23f77cc166bbab435bff3e96520f580c51b66409b227f1db7fdcb2b7 not found: ID does not exist" Aug 04 18:23:54.721318 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.721282 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:54.726280 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726250 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726284 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726298 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hvz\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-kube-api-access-l6hvz\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726312 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726325 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726336 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.726398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.726347 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:23:54.728234 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.728210 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bcb508fd-openshift-default-577df4f76d-pmhkh"] Aug 04 18:23:54.741741 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:23:54.741658 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" path="/var/lib/kubelet/pods/c0f1540c-dc98-4c8c-8fd6-7a1233d442f3/volumes" Aug 04 18:24:00.892440 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:00.892395 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:24:05.786777 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.786738 2559 generic.go:356] "Generic (PLEG): container finished" podID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerID="fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6" exitCode=0 Aug 04 18:24:05.787233 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.786808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerDied","Data":"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6"} Aug 04 18:24:05.787301 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.787247 2559 scope.go:117] "RemoveContainer" containerID="fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6" Aug 04 18:24:05.788372 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.788344 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:24:05.788457 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.788440 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:24:05.788535 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:05.788493 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:24:06.678092 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:06.678053 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:06.794947 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:06.794904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerStarted","Data":"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d"} Aug 04 18:24:07.266389 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:07.266338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:07.267726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:07.267694 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:24:07.267884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:07.267747 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:24:08.266610 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:08.266566 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:24:08.267072 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:08.266640 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:24:09.266322 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:09.266276 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:24:09.266574 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:09.266339 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:24:10.266761 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:10.266716 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:24:10.267184 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:10.266792 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:24:11.266457 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:11.266415 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Aug 04 18:24:11.266669 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:11.266475 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Aug 04 18:24:12.270291 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:12.270258 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:12.270697 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:12.270678 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:12.271309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:12.271292 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:24.397666 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:24.397625 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:24:24.398185 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:24.397865 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" containerID="cri-o://ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d" gracePeriod=30 Aug 04 18:24:26.987914 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:26.987879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:24:27.272696 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:27.272604 2559 patch_prober.go:28] interesting pod/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:24:27.272696 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:27.272678 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:24:29.652936 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.652909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:29.781751 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781649 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.781918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.781918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781797 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.781918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781826 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.781918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781870 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.781918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9q5d\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.781934 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782001 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782034 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token\") pod \"4b8eedb4-c456-46a3-90ab-746bc726270a\" (UID: \"4b8eedb4-c456-46a3-90ab-746bc726270a\") " Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782110 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782129 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:24:29.782183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782150 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:24:29.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782312 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:24:29.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782321 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:24:29.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782404 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782422 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.782476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.782438 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b8eedb4-c456-46a3-90ab-746bc726270a-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.784236 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.784206 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:24:29.784236 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.784212 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:24:29.784397 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.784242 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d" (OuterVolumeSpecName: "kube-api-access-c9q5d") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "kube-api-access-c9q5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:24:29.784397 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.784315 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b8eedb4-c456-46a3-90ab-746bc726270a" (UID: "4b8eedb4-c456-46a3-90ab-746bc726270a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:24:29.883893 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883858 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.883893 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883884 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.883893 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883893 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9q5d\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-kube-api-access-c9q5d\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.883893 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883903 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.884149 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883911 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.884149 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.883918 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b8eedb4-c456-46a3-90ab-746bc726270a-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:24:29.930814 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.930775 2559 generic.go:356] "Generic (PLEG): container finished" podID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerID="ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d" exitCode=0 Aug 04 18:24:29.930995 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.930851 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" Aug 04 18:24:29.930995 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.930869 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerDied","Data":"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d"} Aug 04 18:24:29.930995 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.930928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798" event={"ID":"4b8eedb4-c456-46a3-90ab-746bc726270a","Type":"ContainerDied","Data":"0e42d2c8772794a665ddaa183e3e22da47c135f901f5584ebad5dc11887409ee"} Aug 04 18:24:29.930995 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.930951 2559 scope.go:117] "RemoveContainer" containerID="ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d" Aug 04 18:24:29.947780 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.947762 2559 scope.go:117] "RemoveContainer" containerID="fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6" Aug 04 18:24:29.957006 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.956987 2559 scope.go:117] "RemoveContainer" containerID="ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d" Aug 04 18:24:29.957272 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:24:29.957250 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d\": container with ID starting with ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d not found: ID does not exist" containerID="ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d" Aug 04 18:24:29.957340 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.957282 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d"} err="failed to get container status \"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d\": rpc error: code = NotFound desc = could not find container \"ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d\": container with ID starting with ca0def39edcbc344899690cb2a1bd6e8c99899134447572af0d10f004e8ef95d not found: ID does not exist" Aug 04 18:24:29.957340 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.957300 2559 scope.go:117] "RemoveContainer" containerID="fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6" Aug 04 18:24:29.957605 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:24:29.957587 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6\": container with ID starting with fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6 not found: ID does not exist" containerID="fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6" Aug 04 18:24:29.957684 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.957607 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6"} err="failed to get container status \"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6\": rpc error: code = NotFound desc = could not find container \"fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6\": container with ID starting with fefa2e9f85e66cabb3d139db36bd47cf250c032f0da914769e94d993a9b76de6 not found: ID does not exist" Aug 04 18:24:29.958535 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.958511 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:24:29.962412 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:29.962392 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-07becd73-openshift-default-9b8cf5b44-5q798"] Aug 04 18:24:30.740126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:30.740095 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" path="/var/lib/kubelet/pods/4b8eedb4-c456-46a3-90ab-746bc726270a/volumes" Aug 04 18:24:39.784491 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:24:39.784451 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:26:15.023932 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:15.023893 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:26:26.303034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:26.302984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:26:31.204334 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204286 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204717 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204731 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204744 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204750 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204757 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.204812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204763 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.205024 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204843 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:26:31.205024 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204852 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.205024 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.204857 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8eedb4-c456-46a3-90ab-746bc726270a" containerName="istio-proxy" Aug 04 18:26:31.206971 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.206953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.214488 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.214463 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6abb689a-openshift-default-dockercfg-zjbz9\"" Aug 04 18:26:31.251960 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.251286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:26:31.336660 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.336875 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.336875 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.336875 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.336875 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.337095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.337095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.337095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.336957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.337095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.337018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp7v\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437604 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437824 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437824 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437824 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437998 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437841 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437998 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437893 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.437998 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.437958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp7v\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438156 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438156 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438156 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438156 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438096 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438368 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438439 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.438804 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.438778 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.440279 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.440250 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.440771 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.440749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.449614 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.449584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.451582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.451564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp7v\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v\") pod \"e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.517118 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.517012 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:31.697125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.697089 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:26:31.703034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.702994 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:31.703141 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.703073 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:31.703141 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:31.703101 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:32.655323 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:32.655272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerStarted","Data":"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b"} Aug 04 18:26:32.655753 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:32.655343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerStarted","Data":"b3253d0ab8586cbbbfdf77372cf803a99aef09f8e713683439f4d183f721883e"} Aug 04 18:26:32.681458 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:32.681392 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podStartSLOduration=1.681371156 podStartE2EDuration="1.681371156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:26:32.68029613 +0000 UTC m=+2054.422450191" watchObservedRunningTime="2026-08-04 18:26:32.681371156 +0000 UTC m=+2054.423525216" Aug 04 18:26:33.517576 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:33.517534 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:33.523902 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:33.523871 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:33.660833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:33.660794 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:33.661950 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:33.661924 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:39.285939 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:39.285903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:26:48.683781 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.683730 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:33740->10.133.0.55:15021: read: connection reset by peer" start-of-body= Aug 04 18:26:48.684213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.683850 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:33740->10.133.0.55:15021: read: connection reset by peer" Aug 04 18:26:48.751653 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.751616 2559 generic.go:356] "Generic (PLEG): container finished" podID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerID="707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b" exitCode=0 Aug 04 18:26:48.751852 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.751702 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerDied","Data":"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b"} Aug 04 18:26:48.752227 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.752212 2559 scope.go:117] "RemoveContainer" containerID="707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b" Aug 04 18:26:48.753394 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.753363 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:48.753486 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.753468 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:48.753577 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:48.753541 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:26:49.759923 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:49.759884 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerStarted","Data":"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016"} Aug 04 18:26:50.517743 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:50.517702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:50.518098 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:50.518071 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:26:50.518157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:50.518135 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:26:51.518249 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:51.518206 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:26:51.518708 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:51.518281 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:26:52.518339 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:52.518299 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:26:52.518797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:52.518363 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:26:53.518232 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:53.518192 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:26:53.518428 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:53.518259 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:26:54.517403 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:54.517362 2559 patch_prober.go:28] interesting pod/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Aug 04 18:26:54.517631 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:54.517435 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Aug 04 18:26:55.521685 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:55.521650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:55.522126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:55.522001 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:26:55.522804 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:26:55.522788 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:27:20.594516 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:20.594463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:27:25.920486 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:25.920443 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:27:25.921030 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:25.920782 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" containerID="cri-o://053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016" gracePeriod=30 Aug 04 18:27:31.184694 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.184664 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:27:31.267802 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.267690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.267802 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.267753 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.267802 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.267791 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.267897 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.267970 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzp7v\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268061 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268083 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268113 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268549 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data\") pod \"fc04f3ec-20ee-4848-8a81-cbe49a154651\" (UID: \"fc04f3ec-20ee-4848-8a81-cbe49a154651\") " Aug 04 18:27:31.268549 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268226 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:27:31.268549 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268444 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:27:31.268549 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268455 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:27:31.268794 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268626 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.268794 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268649 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.268794 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268667 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc04f3ec-20ee-4848-8a81-cbe49a154651-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.268794 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268681 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.269004 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.268965 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:27:31.270385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.270358 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:27:31.270545 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.270410 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:27:31.270745 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.270727 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:27:31.270812 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.270759 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v" (OuterVolumeSpecName: "kube-api-access-qzp7v") pod "fc04f3ec-20ee-4848-8a81-cbe49a154651" (UID: "fc04f3ec-20ee-4848-8a81-cbe49a154651"). InnerVolumeSpecName "kube-api-access-qzp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:27:31.370033 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.369993 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzp7v\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-kube-api-access-qzp7v\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.370033 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.370029 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.370033 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.370038 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.370033 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.370046 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:31.370342 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:31.370058 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc04f3ec-20ee-4848-8a81-cbe49a154651-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:27:32.007218 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.007180 2559 generic.go:356] "Generic (PLEG): container finished" podID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerID="053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016" exitCode=0 Aug 04 18:27:32.007398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.007242 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" Aug 04 18:27:32.007398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.007245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerDied","Data":"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016"} Aug 04 18:27:32.007398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.007374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp" event={"ID":"fc04f3ec-20ee-4848-8a81-cbe49a154651","Type":"ContainerDied","Data":"b3253d0ab8586cbbbfdf77372cf803a99aef09f8e713683439f4d183f721883e"} Aug 04 18:27:32.007559 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.007403 2559 scope.go:117] "RemoveContainer" containerID="053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016" Aug 04 18:27:32.017741 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.017718 2559 scope.go:117] "RemoveContainer" containerID="707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b" Aug 04 18:27:32.027085 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.027063 2559 scope.go:117] "RemoveContainer" containerID="053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016" Aug 04 18:27:32.027389 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:27:32.027368 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016\": container with ID starting with 053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016 not found: ID does not exist" containerID="053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016" Aug 04 18:27:32.027439 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.027401 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016"} err="failed to get container status \"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016\": rpc error: code = NotFound desc = could not find container \"053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016\": container with ID starting with 053847e3e12357461b97028bb0a173447363025a7d01f5cbf195480a62cab016 not found: ID does not exist" Aug 04 18:27:32.027439 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.027424 2559 scope.go:117] "RemoveContainer" containerID="707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b" Aug 04 18:27:32.027721 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:27:32.027703 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b\": container with ID starting with 707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b not found: ID does not exist" containerID="707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b" Aug 04 18:27:32.027768 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.027729 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b"} err="failed to get container status \"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b\": rpc error: code = NotFound desc = could not find container \"707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b\": container with ID starting with 707686f353d6e2711e25181c9ebc603b16993763d4c3d2f88eeb623d5da6278b not found: ID does not exist" Aug 04 18:27:32.031432 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.031402 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:27:32.037685 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.037658 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6abb689a-openshift-default-cb7bf5d88-wr9tp"] Aug 04 18:27:32.740150 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:27:32.740116 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" path="/var/lib/kubelet/pods/fc04f3ec-20ee-4848-8a81-cbe49a154651/volumes" Aug 04 18:28:01.788970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:01.788927 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:28:08.325070 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.325033 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:28:08.325535 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.325231 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" podUID="b606e682-00c3-4df9-84a9-1ba2510e97c1" containerName="manager" containerID="cri-o://147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10" gracePeriod=10 Aug 04 18:28:08.599165 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.599137 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:28:08.744778 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.744749 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw7tl\" (UniqueName: \"kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl\") pod \"b606e682-00c3-4df9-84a9-1ba2510e97c1\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " Aug 04 18:28:08.744932 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.744790 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume\") pod \"b606e682-00c3-4df9-84a9-1ba2510e97c1\" (UID: \"b606e682-00c3-4df9-84a9-1ba2510e97c1\") " Aug 04 18:28:08.745147 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.745128 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b606e682-00c3-4df9-84a9-1ba2510e97c1" (UID: "b606e682-00c3-4df9-84a9-1ba2510e97c1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:28:08.746939 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.746908 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl" (OuterVolumeSpecName: "kube-api-access-sw7tl") pod "b606e682-00c3-4df9-84a9-1ba2510e97c1" (UID: "b606e682-00c3-4df9-84a9-1ba2510e97c1"). InnerVolumeSpecName "kube-api-access-sw7tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:28:08.845567 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.845470 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw7tl\" (UniqueName: \"kubernetes.io/projected/b606e682-00c3-4df9-84a9-1ba2510e97c1-kube-api-access-sw7tl\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:28:08.845567 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:08.845514 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b606e682-00c3-4df9-84a9-1ba2510e97c1-extensions-socket-volume\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:28:09.239572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.239462 2559 generic.go:356] "Generic (PLEG): container finished" podID="b606e682-00c3-4df9-84a9-1ba2510e97c1" containerID="147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10" exitCode=0 Aug 04 18:28:09.239572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.239528 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" event={"ID":"b606e682-00c3-4df9-84a9-1ba2510e97c1","Type":"ContainerDied","Data":"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10"} Aug 04 18:28:09.239572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.239535 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" Aug 04 18:28:09.239572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.239569 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz" event={"ID":"b606e682-00c3-4df9-84a9-1ba2510e97c1","Type":"ContainerDied","Data":"8328ce42ab4f6a326788d7cb1cbcc441e371d6037241e35481349cdfc3d73f4f"} Aug 04 18:28:09.239894 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.239591 2559 scope.go:117] "RemoveContainer" containerID="147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10" Aug 04 18:28:09.251083 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.251059 2559 scope.go:117] "RemoveContainer" containerID="147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10" Aug 04 18:28:09.251363 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:28:09.251343 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10\": container with ID starting with 147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10 not found: ID does not exist" containerID="147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10" Aug 04 18:28:09.251403 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.251374 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10"} err="failed to get container status \"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10\": rpc error: code = NotFound desc = could not find container \"147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10\": container with ID starting with 147e022b6b9d45338a2c601f879a7d49ab84744422567af569642d6cc92cae10 not found: ID does not exist" Aug 04 18:28:09.264618 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.264587 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:28:09.268398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:09.268373 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mt8vz"] Aug 04 18:28:10.739415 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:28:10.739339 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b606e682-00c3-4df9-84a9-1ba2510e97c1" path="/var/lib/kubelet/pods/b606e682-00c3-4df9-84a9-1ba2510e97c1/volumes" Aug 04 18:29:23.939964 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.939928 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9"] Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940360 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940378 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b606e682-00c3-4df9-84a9-1ba2510e97c1" containerName="manager" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940385 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b606e682-00c3-4df9-84a9-1ba2510e97c1" containerName="manager" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940391 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940397 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940408 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.940467 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940413 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.940766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940472 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b606e682-00c3-4df9-84a9-1ba2510e97c1" containerName="manager" Aug 04 18:29:23.940766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940479 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.940766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940486 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f1540c-dc98-4c8c-8fd6-7a1233d442f3" containerName="istio-proxy" Aug 04 18:29:23.940766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.940514 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc04f3ec-20ee-4848-8a81-cbe49a154651" containerName="istio-proxy" Aug 04 18:29:23.943607 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.943583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:23.945909 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.945878 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wg58v\"" Aug 04 18:29:23.955223 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:23.955189 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9"] Aug 04 18:29:24.052043 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.052003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7dl\" (UniqueName: \"kubernetes.io/projected/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-kube-api-access-vd7dl\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.052233 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.052060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.152835 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.152795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.153036 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.152935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7dl\" (UniqueName: \"kubernetes.io/projected/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-kube-api-access-vd7dl\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.153275 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.153252 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.163362 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.163333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7dl\" (UniqueName: \"kubernetes.io/projected/537c67f7-4d6b-439a-a1f6-0fa79f4fc42d-kube-api-access-vd7dl\") pod \"kuadrant-operator-controller-manager-68cf589589-wz4z9\" (UID: \"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.254783 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.254617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.406912 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.406886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9"] Aug 04 18:29:24.409314 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:29:24.409285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537c67f7_4d6b_439a_a1f6_0fa79f4fc42d.slice/crio-0a6e23fb22592a6142542ecf6dc7238d2f539f51e79c91af98a2f4bbcf144ec9 WatchSource:0}: Error finding container 0a6e23fb22592a6142542ecf6dc7238d2f539f51e79c91af98a2f4bbcf144ec9: Status 404 returned error can't find the container with id 0a6e23fb22592a6142542ecf6dc7238d2f539f51e79c91af98a2f4bbcf144ec9 Aug 04 18:29:24.658392 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.658360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" event={"ID":"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d","Type":"ContainerStarted","Data":"f0a59a65754beb648e8df9719b38f9cb297c87b513af0c2f1a311217d35cefbf"} Aug 04 18:29:24.658614 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.658412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" event={"ID":"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d","Type":"ContainerStarted","Data":"0a6e23fb22592a6142542ecf6dc7238d2f539f51e79c91af98a2f4bbcf144ec9"} Aug 04 18:29:24.658614 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.658472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:24.695403 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:24.695346 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" podStartSLOduration=1.695326153 podStartE2EDuration="1.695326153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:29:24.693758388 +0000 UTC m=+2226.435912447" watchObservedRunningTime="2026-08-04 18:29:24.695326153 +0000 UTC m=+2226.437480204" Aug 04 18:29:35.667164 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:35.667129 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:29:42.430532 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.430474 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:29:42.434926 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.434897 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.437163 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.437134 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ab5c72f0-openshift-default-dockercfg-6bkw8\"" Aug 04 18:29:42.447575 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.447544 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:29:42.514167 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514374 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514178 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514374 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514267 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514374 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkdg\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514374 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514324 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.514634 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.514493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.615706 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.615951 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.615951 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.615951 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.615951 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.615982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616075 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkdg\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616386 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616230 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616445 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616580 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616543 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616706 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.616776 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.616701 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.618224 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.618203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.618811 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.618792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.626134 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.626103 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.626268 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.626254 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkdg\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg\") pod \"e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.747490 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.747370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:42.903867 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.903837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:29:42.906020 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:29:42.905988 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e930c5_6d9b_483e_bf22_ee7a3c57388d.slice/crio-37828d53caf92b06c7c7e67dd267b534748cbbe942a6c75e5735095c64d3a1da WatchSource:0}: Error finding container 37828d53caf92b06c7c7e67dd267b534748cbbe942a6c75e5735095c64d3a1da: Status 404 returned error can't find the container with id 37828d53caf92b06c7c7e67dd267b534748cbbe942a6c75e5735095c64d3a1da Aug 04 18:29:42.908376 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.908340 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:29:42.908526 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.908431 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:29:42.908526 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:42.908477 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:29:43.776385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:43.776334 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerStarted","Data":"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d"} Aug 04 18:29:43.777113 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:43.776405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerStarted","Data":"37828d53caf92b06c7c7e67dd267b534748cbbe942a6c75e5735095c64d3a1da"} Aug 04 18:29:43.802299 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:43.802238 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podStartSLOduration=1.802220124 podStartE2EDuration="1.802220124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:29:43.801318937 +0000 UTC m=+2245.543472998" watchObservedRunningTime="2026-08-04 18:29:43.802220124 +0000 UTC m=+2245.544374280" Aug 04 18:29:44.694830 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:44.694788 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:29:44.748025 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:44.747989 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:44.753352 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:44.753323 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:44.783003 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:44.782973 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:44.784146 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:44.784123 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:29:55.505222 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:29:55.505179 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:30:00.078979 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.078926 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:53666->10.133.0.57:15021: read: connection reset by peer" start-of-body= Aug 04 18:30:00.079372 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.079023 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:53666->10.133.0.57:15021: read: connection reset by peer" Aug 04 18:30:00.893857 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.893823 2559 generic.go:356] "Generic (PLEG): container finished" podID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerID="aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d" exitCode=0 Aug 04 18:30:00.894089 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.893901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerDied","Data":"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d"} Aug 04 18:30:00.894413 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.894395 2559 scope.go:117] "RemoveContainer" containerID="aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d" Aug 04 18:30:00.895554 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.895528 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:30:00.895625 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.895617 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:30:00.895664 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:00.895652 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:30:01.902607 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:01.902572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerStarted","Data":"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148"} Aug 04 18:30:02.748524 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:02.748472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:30:02.748819 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:02.748782 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:30:02.748900 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:02.748848 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:30:03.748306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:03.748260 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:30:03.748754 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:03.748342 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:30:04.748071 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:04.748032 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:30:04.748278 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:04.748091 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:30:05.748314 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:05.748261 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:30:05.748779 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:05.748343 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:30:06.747833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:06.747791 2559 patch_prober.go:28] interesting pod/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Aug 04 18:30:06.748053 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:06.747854 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Aug 04 18:30:07.752109 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:07.752078 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:30:07.752581 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:07.752456 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:30:07.753327 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:07.753304 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:30:32.785065 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:32.785020 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:30:50.197157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:30:50.197113 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:31:13.889788 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:31:13.889699 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:31:21.584434 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:31:21.584394 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:32:09.232469 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:09.232422 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:32:09.233048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:09.232701 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" containerID="cri-o://e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148" gracePeriod=30 Aug 04 18:32:11.388564 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:11.388508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:32:14.485145 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.485118 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:32:14.616106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616065 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616107 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616128 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616148 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616181 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616222 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616331 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616385 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616379 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnkdg\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616444 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs\") pod \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\" (UID: \"b2e930c5-6d9b-483e-bf22-ee7a3c57388d\") " Aug 04 18:32:14.616760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616442 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:32:14.616760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616461 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:32:14.616760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616654 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:32:14.616760 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616686 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:32:14.617015 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616908 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.617015 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616927 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.617015 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616940 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.617015 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616964 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.617015 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.616936 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:32:14.618806 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.618775 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg" (OuterVolumeSpecName: "kube-api-access-qnkdg") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "kube-api-access-qnkdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:32:14.618947 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.618814 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:32:14.619014 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.618970 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:32:14.619014 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.618991 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2e930c5-6d9b-483e-bf22-ee7a3c57388d" (UID: "b2e930c5-6d9b-483e-bf22-ee7a3c57388d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:32:14.666565 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.666529 2559 generic.go:356] "Generic (PLEG): container finished" podID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerID="e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148" exitCode=0 Aug 04 18:32:14.666742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.666578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerDied","Data":"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148"} Aug 04 18:32:14.666742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.666591 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" Aug 04 18:32:14.666742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.666656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g" event={"ID":"b2e930c5-6d9b-483e-bf22-ee7a3c57388d","Type":"ContainerDied","Data":"37828d53caf92b06c7c7e67dd267b534748cbbe942a6c75e5735095c64d3a1da"} Aug 04 18:32:14.666742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.666680 2559 scope.go:117] "RemoveContainer" containerID="e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148" Aug 04 18:32:14.677088 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.677065 2559 scope.go:117] "RemoveContainer" containerID="aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d" Aug 04 18:32:14.688650 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.688630 2559 scope.go:117] "RemoveContainer" containerID="e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148" Aug 04 18:32:14.688961 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:32:14.688935 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148\": container with ID starting with e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148 not found: ID does not exist" containerID="e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148" Aug 04 18:32:14.689073 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.688968 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148"} err="failed to get container status \"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148\": rpc error: code = NotFound desc = could not find container \"e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148\": container with ID starting with e6a16a37e0a7962a6d367fd60cd2aec30b05406f73c01a1c4633a369a18db148 not found: ID does not exist" Aug 04 18:32:14.689073 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.688993 2559 scope.go:117] "RemoveContainer" containerID="aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d" Aug 04 18:32:14.689252 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:32:14.689236 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d\": container with ID starting with aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d not found: ID does not exist" containerID="aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d" Aug 04 18:32:14.689300 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.689255 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d"} err="failed to get container status \"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d\": rpc error: code = NotFound desc = could not find container \"aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d\": container with ID starting with aab1c125c17dc0a1c6486718c1a43e7dfb8c372ab8fbf73608284d39926b664d not found: ID does not exist" Aug 04 18:32:14.697752 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.697724 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:32:14.702463 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.702435 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ab5c72f0-openshift-default-948787868-l4r4g"] Aug 04 18:32:14.718360 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.718329 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.718360 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.718359 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.718601 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.718371 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnkdg\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-kube-api-access-qnkdg\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.718601 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.718387 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.718601 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.718399 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e930c5-6d9b-483e-bf22-ee7a3c57388d-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:32:14.739907 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:14.739866 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" path="/var/lib/kubelet/pods/b2e930c5-6d9b-483e-bf22-ee7a3c57388d/volumes" Aug 04 18:32:23.390000 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:23.389962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:32:31.897443 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:32:31.897406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:33:17.665154 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665103 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:33:17.665654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665514 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:33:17.665654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665526 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:33:17.665654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665536 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:33:17.665654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665541 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:33:17.665654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.665597 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:33:17.668758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.668742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.671042 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.671019 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c70eb3-openshift-default-dockercfg-gblmx\"" Aug 04 18:33:17.681939 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.681916 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:33:17.731887 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.731846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.731887 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.731887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.731956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.732213 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.732164 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9r7\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833042 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833160 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833273 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9r7\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833582 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.833908 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833881 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.834028 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.834028 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.833976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.834115 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.834067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.835607 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.835590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.835930 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.835905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.843106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.843078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.843412 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.843391 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9r7\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7\") pod \"e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:17.979144 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:17.979062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:18.126115 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:18.126081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:33:18.126339 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:33:18.126310 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3dee87_255b_4149_980a_7712b00a5db0.slice/crio-343b345b38194d7590a85c20f56ced772b3a81bb068e905b78cffe50ede2f706 WatchSource:0}: Error finding container 343b345b38194d7590a85c20f56ced772b3a81bb068e905b78cffe50ede2f706: Status 404 returned error can't find the container with id 343b345b38194d7590a85c20f56ced772b3a81bb068e905b78cffe50ede2f706 Aug 04 18:33:18.128665 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:18.128625 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:18.128766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:18.128708 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:18.128766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:18.128759 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:19.046108 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:19.046073 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerStarted","Data":"581f00ae998da3cb3bccb53908025385a0674f272b276cee641f125ffd67401b"} Aug 04 18:33:19.046538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:19.046135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerStarted","Data":"343b345b38194d7590a85c20f56ced772b3a81bb068e905b78cffe50ede2f706"} Aug 04 18:33:19.070922 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:19.070875 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podStartSLOduration=2.070858988 podStartE2EDuration="2.070858988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:19.068878808 +0000 UTC m=+2460.811032867" watchObservedRunningTime="2026-08-04 18:33:19.070858988 +0000 UTC m=+2460.813013123" Aug 04 18:33:19.980281 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:19.980243 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:19.985182 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:19.985156 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:20.052125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:20.052095 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:20.053186 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:20.053164 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:25.105591 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:25.105555 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:33:35.053741 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.053702 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:35.054173 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.053761 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:35.140302 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.140268 2559 generic.go:356] "Generic (PLEG): container finished" podID="ff3dee87-255b-4149-980a-7712b00a5db0" containerID="581f00ae998da3cb3bccb53908025385a0674f272b276cee641f125ffd67401b" exitCode=0 Aug 04 18:33:35.140534 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.140348 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerDied","Data":"581f00ae998da3cb3bccb53908025385a0674f272b276cee641f125ffd67401b"} Aug 04 18:33:35.140898 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.140882 2559 scope.go:117] "RemoveContainer" containerID="581f00ae998da3cb3bccb53908025385a0674f272b276cee641f125ffd67401b" Aug 04 18:33:35.142151 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.142121 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:35.142250 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.142196 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:35.142250 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:35.142224 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:33:36.157789 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:36.157751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerStarted","Data":"6d9b5ab0ffa9ef9fc826e12b1698c5fbfdf515b8c856cd11d5ac47c019d3e6c2"} Aug 04 18:33:36.980096 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:36.980063 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:36.980279 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:36.980191 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:36.980279 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:36.980235 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:37.980043 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:37.980008 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:37.980430 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:37.980063 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:38.980005 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:38.979964 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:38.980193 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:38.980020 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:39.980396 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:39.980357 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:39.980868 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:39.980422 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:40.979597 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:40.979561 2559 patch_prober.go:28] interesting pod/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Aug 04 18:33:40.979780 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:40.979623 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Aug 04 18:33:41.983668 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:41.983632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:41.984095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:41.983977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:41.984586 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:41.984566 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:33:56.492438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:33:56.492392 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:34:37.392129 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:34:37.392090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:34:47.242778 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:34:47.242738 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:35:14.000892 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:14.000858 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:35:24.296380 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:24.296346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:35:53.907261 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.907227 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc"] Aug 04 18:35:53.907769 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.907716 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e930c5-6d9b-483e-bf22-ee7a3c57388d" containerName="istio-proxy" Aug 04 18:35:53.909884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.909866 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:53.912283 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.912260 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:35:53.912388 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.912300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:35:53.918840 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.918817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc"] Aug 04 18:35:53.980805 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:53.980774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc\" (UID: \"7e6d8020-5f52-4d14-97a7-42c68666f709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:54.081991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.081948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc\" (UID: \"7e6d8020-5f52-4d14-97a7-42c68666f709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:54.082566 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.082547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc\" (UID: \"7e6d8020-5f52-4d14-97a7-42c68666f709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:54.222709 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.222628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:54.361833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.361799 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc"] Aug 04 18:35:54.363914 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:35:54.363885 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6d8020_5f52_4d14_97a7_42c68666f709.slice/crio-7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac WatchSource:0}: Error finding container 7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac: Status 404 returned error can't find the container with id 7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac Aug 04 18:35:54.366221 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.366204 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:35:54.959049 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:54.959014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" event={"ID":"7e6d8020-5f52-4d14-97a7-42c68666f709","Type":"ContainerStarted","Data":"7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac"} Aug 04 18:35:56.411692 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:56.411662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:35:56.971954 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:56.971919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" event={"ID":"7e6d8020-5f52-4d14-97a7-42c68666f709","Type":"ContainerStarted","Data":"ac600a4c471eb070b8e40183061566b7d3cd3f89ca0c5ab1f2bf816485152e29"} Aug 04 18:35:56.991530 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:56.991463 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" podStartSLOduration=1.691702636 podStartE2EDuration="3.991445996s" totalImagesPullingTime="2.29974336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:56.989556649 +0000 UTC m=+2618.731710709" watchObservedRunningTime="2026-08-04 18:35:56.991445996 +0000 UTC m=+2618.733600054" Aug 04 18:35:57.978511 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:57.978476 2559 generic.go:356] "Generic (PLEG): container finished" podID="7e6d8020-5f52-4d14-97a7-42c68666f709" containerID="ac600a4c471eb070b8e40183061566b7d3cd3f89ca0c5ab1f2bf816485152e29" exitCode=0 Aug 04 18:35:57.978948 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:57.978548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" event={"ID":"7e6d8020-5f52-4d14-97a7-42c68666f709","Type":"ContainerDied","Data":"ac600a4c471eb070b8e40183061566b7d3cd3f89ca0c5ab1f2bf816485152e29"} Aug 04 18:35:59.122893 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.122867 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:59.245884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.245787 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca\") pod \"7e6d8020-5f52-4d14-97a7-42c68666f709\" (UID: \"7e6d8020-5f52-4d14-97a7-42c68666f709\") " Aug 04 18:35:59.246152 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.246125 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e6d8020-5f52-4d14-97a7-42c68666f709" (UID: "7e6d8020-5f52-4d14-97a7-42c68666f709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:35:59.347565 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.347526 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6d8020-5f52-4d14-97a7-42c68666f709-maas-api-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:35:59.990695 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.990661 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" Aug 04 18:35:59.990867 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.990694 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc" event={"ID":"7e6d8020-5f52-4d14-97a7-42c68666f709","Type":"ContainerDied","Data":"7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac"} Aug 04 18:35:59.990867 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:35:59.990740 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b6fbc39f02805e103068c08d0d627fdbe6eb3b398dda49bb841d9f1af82e8ac" Aug 04 18:36:05.038664 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:05.038629 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc"] Aug 04 18:36:05.040915 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:05.040894 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c70eb3-l9dkc"] Aug 04 18:36:06.740198 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:06.740165 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e6d8020-5f52-4d14-97a7-42c68666f709" path="/var/lib/kubelet/pods/7e6d8020-5f52-4d14-97a7-42c68666f709/volumes" Aug 04 18:36:07.199911 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:07.199879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:36:12.907686 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:12.907650 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:36:12.908084 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:12.907868 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" containerID="cri-o://6d9b5ab0ffa9ef9fc826e12b1698c5fbfdf515b8c856cd11d5ac47c019d3e6c2" gracePeriod=30 Aug 04 18:36:18.103458 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.103410 2559 generic.go:356] "Generic (PLEG): container finished" podID="ff3dee87-255b-4149-980a-7712b00a5db0" containerID="6d9b5ab0ffa9ef9fc826e12b1698c5fbfdf515b8c856cd11d5ac47c019d3e6c2" exitCode=0 Aug 04 18:36:18.103917 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.103587 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerDied","Data":"6d9b5ab0ffa9ef9fc826e12b1698c5fbfdf515b8c856cd11d5ac47c019d3e6c2"} Aug 04 18:36:18.103917 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.103669 2559 scope.go:117] "RemoveContainer" containerID="581f00ae998da3cb3bccb53908025385a0674f272b276cee641f125ffd67401b" Aug 04 18:36:18.165023 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.165000 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:36:18.214884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.214799 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.214884 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.214845 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.214888 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.214956 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215015 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215125 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215151 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp9r7\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7\") pod \"ff3dee87-255b-4149-980a-7712b00a5db0\" (UID: \"ff3dee87-255b-4149-980a-7712b00a5db0\") " Aug 04 18:36:18.215361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215162 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:18.215361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215288 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:18.215361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:18.215632 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215427 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:36:18.215693 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215649 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:18.215755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215690 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.215755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215710 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.215755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215722 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.215755 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.215733 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3dee87-255b-4149-980a-7712b00a5db0-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.217635 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.217604 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:36:18.217635 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.217629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:18.217781 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.217649 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7" (OuterVolumeSpecName: "kube-api-access-sp9r7") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "kube-api-access-sp9r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:18.217960 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.217943 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff3dee87-255b-4149-980a-7712b00a5db0" (UID: "ff3dee87-255b-4149-980a-7712b00a5db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:18.316821 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.316785 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.316821 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.316811 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp9r7\" (UniqueName: \"kubernetes.io/projected/ff3dee87-255b-4149-980a-7712b00a5db0-kube-api-access-sp9r7\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.316821 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.316821 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.316821 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.316830 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3dee87-255b-4149-980a-7712b00a5db0-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:18.317101 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:18.316838 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3dee87-255b-4149-980a-7712b00a5db0-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:36:19.110935 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:19.110906 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" Aug 04 18:36:19.111416 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:19.110949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m" event={"ID":"ff3dee87-255b-4149-980a-7712b00a5db0","Type":"ContainerDied","Data":"343b345b38194d7590a85c20f56ced772b3a81bb068e905b78cffe50ede2f706"} Aug 04 18:36:19.111416 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:19.111016 2559 scope.go:117] "RemoveContainer" containerID="6d9b5ab0ffa9ef9fc826e12b1698c5fbfdf515b8c856cd11d5ac47c019d3e6c2" Aug 04 18:36:19.132004 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:19.131981 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:36:19.137694 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:19.137669 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c70eb3-openshift-default-5bdff86cdb-bwp4m"] Aug 04 18:36:20.740052 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:20.740018 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" path="/var/lib/kubelet/pods/ff3dee87-255b-4149-980a-7712b00a5db0/volumes" Aug 04 18:36:20.997917 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:20.997831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:36:27.402174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402141 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:36:27.402588 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402571 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e6d8020-5f52-4d14-97a7-42c68666f709" containerName="revoke-keys" Aug 04 18:36:27.402588 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402584 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e6d8020-5f52-4d14-97a7-42c68666f709" containerName="revoke-keys" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402598 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402604 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402622 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402628 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402683 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402691 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402691 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff3dee87-255b-4149-980a-7712b00a5db0" containerName="istio-proxy" Aug 04 18:36:27.402888 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.402699 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e6d8020-5f52-4d14-97a7-42c68666f709" containerName="revoke-keys" Aug 04 18:36:27.405717 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.405699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.408106 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.408086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a08d36-openshift-default-dockercfg-g9v96\"" Aug 04 18:36:27.420147 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.420123 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:36:27.506202 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506171 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506202 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506296 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbr8b\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506351 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506676 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506676 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.506676 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.506623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607814 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.607991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.607967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608294 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608294 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbr8b\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608294 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608294 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608663 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608856 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608856 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608734 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.608980 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.608923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.610384 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.610363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.610613 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.610596 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.619094 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.619073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbr8b\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.619315 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.619293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token\") pod \"e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.716545 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.716439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:27.866852 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.864267 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:36:27.867586 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:36:27.867548 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ab1dab_24a7_4e1d_b48c_ffab1359bd04.slice/crio-2927494513ead85b4ee89d8b9e3b7930506b8de22a1f8eff534ed975a25a7191 WatchSource:0}: Error finding container 2927494513ead85b4ee89d8b9e3b7930506b8de22a1f8eff534ed975a25a7191: Status 404 returned error can't find the container with id 2927494513ead85b4ee89d8b9e3b7930506b8de22a1f8eff534ed975a25a7191 Aug 04 18:36:27.870159 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.870129 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:27.870220 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.870194 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:27.870253 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:27.870220 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:28.163858 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:28.163823 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerStarted","Data":"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914"} Aug 04 18:36:28.164034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:28.163883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerStarted","Data":"2927494513ead85b4ee89d8b9e3b7930506b8de22a1f8eff534ed975a25a7191"} Aug 04 18:36:28.193845 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:28.193791 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podStartSLOduration=1.19377513 podStartE2EDuration="1.19377513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:28.19309976 +0000 UTC m=+2649.935253830" watchObservedRunningTime="2026-08-04 18:36:28.19377513 +0000 UTC m=+2649.935929189" Aug 04 18:36:28.716575 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:28.716539 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:28.721481 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:28.721456 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:29.169529 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:29.169466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:29.170469 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:29.170447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:34.003957 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:34.003918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:36:44.171330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.171290 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:44.171847 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.171355 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:44.263408 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.263374 2559 generic.go:356] "Generic (PLEG): container finished" podID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerID="037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914" exitCode=0 Aug 04 18:36:44.263597 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.263443 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerDied","Data":"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914"} Aug 04 18:36:44.263881 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.263867 2559 scope.go:117] "RemoveContainer" containerID="037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914" Aug 04 18:36:44.264937 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.264911 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:44.264998 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.264976 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:44.265036 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:44.265007 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:36:45.271427 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:45.271386 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerStarted","Data":"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1"} Aug 04 18:36:45.717056 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:45.717020 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:45.717342 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:45.717310 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:45.717463 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:45.717364 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:46.717558 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:46.717521 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:46.717941 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:46.717580 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:47.717739 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:47.717702 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:47.718131 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:47.717756 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:48.716909 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:48.716872 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:48.717083 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:48.716931 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:49.717849 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:49.717814 2559 patch_prober.go:28] interesting pod/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Aug 04 18:36:49.718238 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:49.717888 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Aug 04 18:36:50.720714 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:50.720671 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:50.721097 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:50.721040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:36:50.721874 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:36:50.721856 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:37:03.398048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:03.398002 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:37:49.680447 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:49.680411 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:37:54.622079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:54.622040 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:37:54.622677 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:54.622317 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" containerID="cri-o://1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1" gracePeriod=30 Aug 04 18:37:59.880239 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.880215 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:37:59.984208 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984126 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984208 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984171 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984208 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984200 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984268 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984303 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984376 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbr8b\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984440 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984474 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo\") pod \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\" (UID: \"21ab1dab-24a7-4e1d-b48c-ffab1359bd04\") " Aug 04 18:37:59.984757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:37:59.984757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:37:59.984757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984686 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:37:59.984757 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984718 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data" (OuterVolumeSpecName: "istio-data") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:37:59.985041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:37:59.985041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984844 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:37:59.985041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984862 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:37:59.985041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984878 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:37:59.985041 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.984893 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:37:59.986580 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.986551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token" (OuterVolumeSpecName: "istio-token") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:37:59.986673 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.986569 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:37:59.986673 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.986632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b" (OuterVolumeSpecName: "kube-api-access-bbr8b") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "kube-api-access-bbr8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:37:59.986749 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:37:59.986723 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21ab1dab-24a7-4e1d-b48c-ffab1359bd04" (UID: "21ab1dab-24a7-4e1d-b48c-ffab1359bd04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:38:00.085628 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.085596 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:38:00.085628 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.085622 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbr8b\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-kube-api-access-bbr8b\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:38:00.085628 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.085633 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:38:00.085855 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.085641 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:38:00.085855 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.085649 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21ab1dab-24a7-4e1d-b48c-ffab1359bd04-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:38:00.703873 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.703838 2559 generic.go:356] "Generic (PLEG): container finished" podID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerID="1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1" exitCode=0 Aug 04 18:38:00.704060 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.703892 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" Aug 04 18:38:00.704060 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.703922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerDied","Data":"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1"} Aug 04 18:38:00.704060 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.703977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2" event={"ID":"21ab1dab-24a7-4e1d-b48c-ffab1359bd04","Type":"ContainerDied","Data":"2927494513ead85b4ee89d8b9e3b7930506b8de22a1f8eff534ed975a25a7191"} Aug 04 18:38:00.704060 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.704000 2559 scope.go:117] "RemoveContainer" containerID="1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1" Aug 04 18:38:00.714142 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.714125 2559 scope.go:117] "RemoveContainer" containerID="037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914" Aug 04 18:38:00.722436 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.722420 2559 scope.go:117] "RemoveContainer" containerID="1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1" Aug 04 18:38:00.722703 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:38:00.722685 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1\": container with ID starting with 1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1 not found: ID does not exist" containerID="1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1" Aug 04 18:38:00.722805 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.722712 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1"} err="failed to get container status \"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1\": rpc error: code = NotFound desc = could not find container \"1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1\": container with ID starting with 1b4807322001d60611e819676d33d7e782f7d2307d93a4cbe81689252cb5c8d1 not found: ID does not exist" Aug 04 18:38:00.722805 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.722729 2559 scope.go:117] "RemoveContainer" containerID="037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914" Aug 04 18:38:00.723006 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:38:00.722988 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914\": container with ID starting with 037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914 not found: ID does not exist" containerID="037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914" Aug 04 18:38:00.723062 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.723017 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914"} err="failed to get container status \"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914\": rpc error: code = NotFound desc = could not find container \"037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914\": container with ID starting with 037287dc2b9f0211e5cf3120ae2d68c85d3a9573b2ea211a1296fb5f3f3da914 not found: ID does not exist" Aug 04 18:38:00.727514 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.727477 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:38:00.732331 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.732306 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a08d36-openshift-default-7b5f6f9b4-g69d2"] Aug 04 18:38:00.739355 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:00.739332 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" path="/var/lib/kubelet/pods/21ab1dab-24a7-4e1d-b48c-ffab1359bd04/volumes" Aug 04 18:38:12.594747 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:12.594708 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:38:25.791445 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:25.791405 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:38:47.621334 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.621299 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:38:47.621753 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.621708 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:38:47.621753 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.621722 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:38:47.621828 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.621823 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:38:47.621868 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.621832 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:38:47.624842 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.624824 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.633861 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.633838 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bc18f1-openshift-default-dockercfg-pmvgr\"" Aug 04 18:38:47.655803 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.655775 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:38:47.733603 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.733568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734022 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.733989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js752\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734142 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734142 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734250 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734160 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734250 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734250 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734481 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.734481 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.734378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835453 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835418 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835453 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835902 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.835962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.836127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.836127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.835979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.836127 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.836018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js752\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.836287 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.836158 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.836287 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.836227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.838217 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.838194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.838421 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.838403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.855771 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.855748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js752\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.857822 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.857799 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token\") pod \"e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:47.935275 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:47.935200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:48.087683 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.087656 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:38:48.089216 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:38:48.089191 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687504cf_df66_4d4b_950f_1bbabacbc214.slice/crio-2b23e3287c42a90d5d54f15442e16bfa1dd29f7933df4f000b8c9b3c99848b91 WatchSource:0}: Error finding container 2b23e3287c42a90d5d54f15442e16bfa1dd29f7933df4f000b8c9b3c99848b91: Status 404 returned error can't find the container with id 2b23e3287c42a90d5d54f15442e16bfa1dd29f7933df4f000b8c9b3c99848b91 Aug 04 18:38:48.091389 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.091360 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:38:48.091441 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.091424 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:38:48.091476 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.091453 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:38:48.986198 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.986164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerStarted","Data":"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b"} Aug 04 18:38:48.986625 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:48.986215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerStarted","Data":"2b23e3287c42a90d5d54f15442e16bfa1dd29f7933df4f000b8c9b3c99848b91"} Aug 04 18:38:49.007639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:49.007588 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podStartSLOduration=2.007574115 podStartE2EDuration="2.007574115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:49.006832917 +0000 UTC m=+2790.748986978" watchObservedRunningTime="2026-08-04 18:38:49.007574115 +0000 UTC m=+2790.749728173" Aug 04 18:38:49.935742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:49.935700 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:49.940572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:49.940547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:49.991368 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:49.991338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:49.992210 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:49.992191 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:38:54.890750 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:38:54.890714 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:39:04.992415 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:04.992379 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:04.992874 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:04.992438 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:05.100616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.100587 2559 generic.go:356] "Generic (PLEG): container finished" podID="687504cf-df66-4d4b-950f-1bbabacbc214" containerID="eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b" exitCode=0 Aug 04 18:39:05.100785 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.100672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerDied","Data":"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b"} Aug 04 18:39:05.101141 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.101125 2559 scope.go:117] "RemoveContainer" containerID="eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b" Aug 04 18:39:05.102135 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.102113 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:39:05.102188 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.102172 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:39:05.102232 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:05.102221 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:39:06.107997 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:06.107959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerStarted","Data":"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee"} Aug 04 18:39:06.935624 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:06.935582 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:39:06.936274 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:06.935960 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:06.936274 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:06.936022 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:07.936613 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:07.936576 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:07.937187 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:07.937155 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:08.936310 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:08.936249 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:08.936650 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:08.936618 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:09.935702 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:09.935658 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:09.935920 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:09.935727 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:10.935926 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:10.935881 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Aug 04 18:39:10.936341 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:10.935949 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Aug 04 18:39:11.940474 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:11.940437 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:39:11.940874 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:11.940799 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:39:11.941437 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:11.941422 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:39:28.592787 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:28.592752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:39:43.202594 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:39:43.202551 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:40:05.997249 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:40:05.997215 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:40:21.484103 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:40:21.484017 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:40:33.692859 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:40:33.692821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:40:48.119320 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:40:48.119277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:40:59.398387 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:40:59.398354 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:41:15.001633 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:41:15.001597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:41:26.213091 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:41:26.213048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:41:41.398007 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:41:41.397923 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:41:52.334022 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:41:52.333963 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:42:08.581307 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:08.581268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:42:19.074689 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:19.074657 2559 scope.go:117] "RemoveContainer" containerID="ac600a4c471eb070b8e40183061566b7d3cd3f89ca0c5ab1f2bf816485152e29" Aug 04 18:42:20.683474 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:20.683440 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:42:34.900901 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:34.900865 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:42:45.083896 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:45.083861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:42:53.195809 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:42:53.195773 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:43:08.987445 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:43:08.987401 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:43:47.416851 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:43:47.416812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:43:59.606651 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:43:59.606615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:44:12.197829 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:44:12.197790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:44:21.289664 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:44:21.289624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:44:41.996621 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:44:41.996543 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:45:09.490796 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:45:09.490753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:45:46.615542 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:45:46.615475 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:45:55.101912 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:45:55.101873 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:46:11.119956 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:11.119870 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:46:18.302242 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.302210 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071572400"] Aug 04 18:46:18.302738 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.302604 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:46:18.302738 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.302616 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ab1dab-24a7-4e1d-b48c-ffab1359bd04" containerName="istio-proxy" Aug 04 18:46:18.305846 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.305830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:18.308776 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.308753 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2twlp\"" Aug 04 18:46:18.315527 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.315486 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071572400"] Aug 04 18:46:18.398456 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.398413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mcs\" (UniqueName: \"kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs\") pod \"test-curl-7546-140211071572400\" (UID: \"55864951-d29b-4779-8d85-8c13ac9cdf66\") " pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:18.499430 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.499392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mcs\" (UniqueName: \"kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs\") pod \"test-curl-7546-140211071572400\" (UID: \"55864951-d29b-4779-8d85-8c13ac9cdf66\") " pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:18.513978 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.513944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mcs\" (UniqueName: \"kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs\") pod \"test-curl-7546-140211071572400\" (UID: \"55864951-d29b-4779-8d85-8c13ac9cdf66\") " pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:18.619024 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.618946 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2twlp\"" Aug 04 18:46:18.626779 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.626756 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:18.773834 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.773796 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071572400"] Aug 04 18:46:18.776620 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:18.776591 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55864951_d29b_4779_8d85_8c13ac9cdf66.slice/crio-fb2da943227213acb2a5ed293e812c696668ab6b18fe6f3629384aa9df0ebf58 WatchSource:0}: Error finding container fb2da943227213acb2a5ed293e812c696668ab6b18fe6f3629384aa9df0ebf58: Status 404 returned error can't find the container with id fb2da943227213acb2a5ed293e812c696668ab6b18fe6f3629384aa9df0ebf58 Aug 04 18:46:18.777898 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:18.777880 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:46:19.570698 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:19.570660 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071572400" event={"ID":"55864951-d29b-4779-8d85-8c13ac9cdf66","Type":"ContainerStarted","Data":"eb53aa995e7637a3a60725ddd3fb38dc802baa90959dd12be6889c9f32922b34"} Aug 04 18:46:19.571104 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:19.570709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071572400" event={"ID":"55864951-d29b-4779-8d85-8c13ac9cdf66","Type":"ContainerStarted","Data":"fb2da943227213acb2a5ed293e812c696668ab6b18fe6f3629384aa9df0ebf58"} Aug 04 18:46:19.713631 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:19.713604 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071572400_55864951-d29b-4779-8d85-8c13ac9cdf66/test-curl-7546-140211071572400/0.log" Aug 04 18:46:20.451185 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:20.451125 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7546-140211071572400" podStartSLOduration=1.750495559 podStartE2EDuration="2.451106397s" totalImagesPullingTime="700.610838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:19.590007594 +0000 UTC m=+3241.332161653" watchObservedRunningTime="2026-08-04 18:46:20.451106397 +0000 UTC m=+3242.193260455" Aug 04 18:46:20.451403 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:20.451269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:46:20.578382 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:20.578338 2559 generic.go:356] "Generic (PLEG): container finished" podID="55864951-d29b-4779-8d85-8c13ac9cdf66" containerID="eb53aa995e7637a3a60725ddd3fb38dc802baa90959dd12be6889c9f32922b34" exitCode=0 Aug 04 18:46:20.578841 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:20.578396 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071572400" event={"ID":"55864951-d29b-4779-8d85-8c13ac9cdf66","Type":"ContainerDied","Data":"eb53aa995e7637a3a60725ddd3fb38dc802baa90959dd12be6889c9f32922b34"} Aug 04 18:46:21.717927 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.717899 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:21.756006 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.755972 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071572400"] Aug 04 18:46:21.760850 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.760823 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071572400"] Aug 04 18:46:21.830935 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.830896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6mcs\" (UniqueName: \"kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs\") pod \"55864951-d29b-4779-8d85-8c13ac9cdf66\" (UID: \"55864951-d29b-4779-8d85-8c13ac9cdf66\") " Aug 04 18:46:21.833534 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.833474 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs" (OuterVolumeSpecName: "kube-api-access-x6mcs") pod "55864951-d29b-4779-8d85-8c13ac9cdf66" (UID: "55864951-d29b-4779-8d85-8c13ac9cdf66"). InnerVolumeSpecName "kube-api-access-x6mcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:21.882156 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.882121 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071570096"] Aug 04 18:46:21.882537 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.882524 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55864951-d29b-4779-8d85-8c13ac9cdf66" containerName="test-curl-7546-140211071572400" Aug 04 18:46:21.882589 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.882539 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="55864951-d29b-4779-8d85-8c13ac9cdf66" containerName="test-curl-7546-140211071572400" Aug 04 18:46:21.882623 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.882593 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="55864951-d29b-4779-8d85-8c13ac9cdf66" containerName="test-curl-7546-140211071572400" Aug 04 18:46:21.885639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.885620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:21.892654 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.892631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071570096"] Aug 04 18:46:21.932216 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:21.932181 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mcs\" (UniqueName: \"kubernetes.io/projected/55864951-d29b-4779-8d85-8c13ac9cdf66-kube-api-access-x6mcs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:22.033368 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.033322 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2lg\" (UniqueName: \"kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg\") pod \"test-curl-7546-140211071570096\" (UID: \"67ccec51-062e-44d8-b702-32bbde82d3a2\") " pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:22.134280 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.134187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2lg\" (UniqueName: \"kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg\") pod \"test-curl-7546-140211071570096\" (UID: \"67ccec51-062e-44d8-b702-32bbde82d3a2\") " pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:22.143575 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.143479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2lg\" (UniqueName: \"kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg\") pod \"test-curl-7546-140211071570096\" (UID: \"67ccec51-062e-44d8-b702-32bbde82d3a2\") " pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:22.196276 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.196232 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:22.347068 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.347041 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071570096"] Aug 04 18:46:22.348612 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:22.348582 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ccec51_062e_44d8_b702_32bbde82d3a2.slice/crio-0180029af3f05b62dc0f5aafb3d1a71e9201f7225b0a891ba11aa7019271f65b WatchSource:0}: Error finding container 0180029af3f05b62dc0f5aafb3d1a71e9201f7225b0a891ba11aa7019271f65b: Status 404 returned error can't find the container with id 0180029af3f05b62dc0f5aafb3d1a71e9201f7225b0a891ba11aa7019271f65b Aug 04 18:46:22.592565 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.592523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071570096" event={"ID":"67ccec51-062e-44d8-b702-32bbde82d3a2","Type":"ContainerStarted","Data":"0180029af3f05b62dc0f5aafb3d1a71e9201f7225b0a891ba11aa7019271f65b"} Aug 04 18:46:22.593810 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.593789 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2da943227213acb2a5ed293e812c696668ab6b18fe6f3629384aa9df0ebf58" Aug 04 18:46:22.594075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.593817 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:22.741683 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:22.741650 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55864951-d29b-4779-8d85-8c13ac9cdf66" path="/var/lib/kubelet/pods/55864951-d29b-4779-8d85-8c13ac9cdf66/volumes" Aug 04 18:46:22.919272 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:22.919168 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071572400\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="55864951-d29b-4779-8d85-8c13ac9cdf66" pod="opendatahub/test-curl-7546-140211071572400" Aug 04 18:46:23.601537 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:23.601346 2559 generic.go:356] "Generic (PLEG): container finished" podID="67ccec51-062e-44d8-b702-32bbde82d3a2" containerID="294a5ab0e8156c647a37c46aa70488cb7b179376b125bd1490cad8342957bf21" exitCode=0 Aug 04 18:46:23.601537 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:23.601471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071570096" event={"ID":"67ccec51-062e-44d8-b702-32bbde82d3a2","Type":"ContainerDied","Data":"294a5ab0e8156c647a37c46aa70488cb7b179376b125bd1490cad8342957bf21"} Aug 04 18:46:24.746465 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.746436 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:24.775123 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.775045 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071570096_67ccec51-062e-44d8-b702-32bbde82d3a2/test-curl-7546-140211071570096/0.log" Aug 04 18:46:24.800269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.800232 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071570096"] Aug 04 18:46:24.807143 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.807109 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071570096"] Aug 04 18:46:24.857734 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.857630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr2lg\" (UniqueName: \"kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg\") pod \"67ccec51-062e-44d8-b702-32bbde82d3a2\" (UID: \"67ccec51-062e-44d8-b702-32bbde82d3a2\") " Aug 04 18:46:24.859897 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.859871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg" (OuterVolumeSpecName: "kube-api-access-mr2lg") pod "67ccec51-062e-44d8-b702-32bbde82d3a2" (UID: "67ccec51-062e-44d8-b702-32bbde82d3a2"). InnerVolumeSpecName "kube-api-access-mr2lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:24.958894 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.958852 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr2lg\" (UniqueName: \"kubernetes.io/projected/67ccec51-062e-44d8-b702-32bbde82d3a2-kube-api-access-mr2lg\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:24.965217 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.965184 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:24.965609 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.965594 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" containerName="test-curl-7546-140211071570096" Aug 04 18:46:24.965660 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.965611 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" containerName="test-curl-7546-140211071570096" Aug 04 18:46:24.965704 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.965688 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" containerName="test-curl-7546-140211071570096" Aug 04 18:46:24.968844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.968820 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:24.974684 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:24.974651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:25.060355 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.060314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m768h\" (UniqueName: \"kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h\") pod \"test-curl-7546-140211071667488\" (UID: \"aad9708d-2fc2-4295-b817-b0aebc8ce959\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:25.161264 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.161157 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m768h\" (UniqueName: \"kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h\") pod \"test-curl-7546-140211071667488\" (UID: \"aad9708d-2fc2-4295-b817-b0aebc8ce959\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:25.171447 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.171416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m768h\" (UniqueName: \"kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h\") pod \"test-curl-7546-140211071667488\" (UID: \"aad9708d-2fc2-4295-b817-b0aebc8ce959\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:25.280018 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.279984 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:25.428741 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.428712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:25.430738 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:25.430710 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad9708d_2fc2_4295_b817_b0aebc8ce959.slice/crio-5cbd6f6e3661ddd9dc178c445eff3fb019db28ca0a5ded26ace8e54597ad8c65 WatchSource:0}: Error finding container 5cbd6f6e3661ddd9dc178c445eff3fb019db28ca0a5ded26ace8e54597ad8c65: Status 404 returned error can't find the container with id 5cbd6f6e3661ddd9dc178c445eff3fb019db28ca0a5ded26ace8e54597ad8c65 Aug 04 18:46:25.613936 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.613906 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:25.613936 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.613916 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0180029af3f05b62dc0f5aafb3d1a71e9201f7225b0a891ba11aa7019271f65b" Aug 04 18:46:25.615204 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:25.615176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071667488" event={"ID":"aad9708d-2fc2-4295-b817-b0aebc8ce959","Type":"ContainerStarted","Data":"5cbd6f6e3661ddd9dc178c445eff3fb019db28ca0a5ded26ace8e54597ad8c65"} Aug 04 18:46:25.626072 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:25.626035 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071570096\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:26.623190 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:26.623158 2559 generic.go:356] "Generic (PLEG): container finished" podID="aad9708d-2fc2-4295-b817-b0aebc8ce959" containerID="234fae8fa4521170039b73405ec4b3b45c5e3ee8c993afd8389ee81c773c9571" exitCode=0 Aug 04 18:46:26.623672 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:26.623248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071667488" event={"ID":"aad9708d-2fc2-4295-b817-b0aebc8ce959","Type":"ContainerDied","Data":"234fae8fa4521170039b73405ec4b3b45c5e3ee8c993afd8389ee81c773c9571"} Aug 04 18:46:26.650759 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:26.650714 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071570096\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" pod="opendatahub/test-curl-7546-140211071570096" Aug 04 18:46:26.741442 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:26.741405 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ccec51-062e-44d8-b702-32bbde82d3a2" path="/var/lib/kubelet/pods/67ccec51-062e-44d8-b702-32bbde82d3a2/volumes" Aug 04 18:46:27.767204 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.767178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:27.793778 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.793749 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071667488_aad9708d-2fc2-4295-b817-b0aebc8ce959/test-curl-7546-140211071667488/0.log" Aug 04 18:46:27.819332 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.819298 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:27.824447 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.824414 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:27.886935 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.886839 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m768h\" (UniqueName: \"kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h\") pod \"aad9708d-2fc2-4295-b817-b0aebc8ce959\" (UID: \"aad9708d-2fc2-4295-b817-b0aebc8ce959\") " Aug 04 18:46:27.889075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.889049 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h" (OuterVolumeSpecName: "kube-api-access-m768h") pod "aad9708d-2fc2-4295-b817-b0aebc8ce959" (UID: "aad9708d-2fc2-4295-b817-b0aebc8ce959"). InnerVolumeSpecName "kube-api-access-m768h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:27.947535 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.947479 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:27.947947 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.947933 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad9708d-2fc2-4295-b817-b0aebc8ce959" containerName="test-curl-7546-140211071667488" Aug 04 18:46:27.947993 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.947950 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad9708d-2fc2-4295-b817-b0aebc8ce959" containerName="test-curl-7546-140211071667488" Aug 04 18:46:27.948042 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.948032 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad9708d-2fc2-4295-b817-b0aebc8ce959" containerName="test-curl-7546-140211071667488" Aug 04 18:46:27.951170 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.951152 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:27.959520 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.959476 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:27.988206 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:27.988164 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m768h\" (UniqueName: \"kubernetes.io/projected/aad9708d-2fc2-4295-b817-b0aebc8ce959-kube-api-access-m768h\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:28.089655 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.089612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v25w4\" (UniqueName: \"kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4\") pod \"test-curl-7546-140211071667488\" (UID: \"66a301ff-638b-4cd5-812b-2aae06b8fbcc\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:28.191023 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.190932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v25w4\" (UniqueName: \"kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4\") pod \"test-curl-7546-140211071667488\" (UID: \"66a301ff-638b-4cd5-812b-2aae06b8fbcc\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:28.201688 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.201653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v25w4\" (UniqueName: \"kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4\") pod \"test-curl-7546-140211071667488\" (UID: \"66a301ff-638b-4cd5-812b-2aae06b8fbcc\") " pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:28.261766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.261720 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:28.412867 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.412837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:28.413947 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:28.413914 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66a301ff_638b_4cd5_812b_2aae06b8fbcc.slice/crio-bb9ad69d427da920a864baaa86fb36dc25f4ff82ae4f52e7e0aaa160094a6b14 WatchSource:0}: Error finding container bb9ad69d427da920a864baaa86fb36dc25f4ff82ae4f52e7e0aaa160094a6b14: Status 404 returned error can't find the container with id bb9ad69d427da920a864baaa86fb36dc25f4ff82ae4f52e7e0aaa160094a6b14 Aug 04 18:46:28.636288 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.636256 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:28.636455 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.636306 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbd6f6e3661ddd9dc178c445eff3fb019db28ca0a5ded26ace8e54597ad8c65" Aug 04 18:46:28.637967 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.637916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071667488" event={"ID":"66a301ff-638b-4cd5-812b-2aae06b8fbcc","Type":"ContainerStarted","Data":"bb9ad69d427da920a864baaa86fb36dc25f4ff82ae4f52e7e0aaa160094a6b14"} Aug 04 18:46:28.651710 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.651679 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7546-140211071667488" oldPodUID="aad9708d-2fc2-4295-b817-b0aebc8ce959" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" Aug 04 18:46:28.740611 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:28.740587 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad9708d-2fc2-4295-b817-b0aebc8ce959" path="/var/lib/kubelet/pods/aad9708d-2fc2-4295-b817-b0aebc8ce959/volumes" Aug 04 18:46:29.645994 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:29.645956 2559 generic.go:356] "Generic (PLEG): container finished" podID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" containerID="d5f05db478a80cffd396d6356893a86465be5557f4d6971d1f12729743f817d2" exitCode=0 Aug 04 18:46:29.646426 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:29.646011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071667488" event={"ID":"66a301ff-638b-4cd5-812b-2aae06b8fbcc","Type":"ContainerDied","Data":"d5f05db478a80cffd396d6356893a86465be5557f4d6971d1f12729743f817d2"} Aug 04 18:46:30.793420 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.793392 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:30.822566 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.822459 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071667488_66a301ff-638b-4cd5-812b-2aae06b8fbcc/test-curl-7546-140211071667488/0.log" Aug 04 18:46:30.858310 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.858277 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:30.863322 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.863290 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071667488"] Aug 04 18:46:30.919421 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.919327 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v25w4\" (UniqueName: \"kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4\") pod \"66a301ff-638b-4cd5-812b-2aae06b8fbcc\" (UID: \"66a301ff-638b-4cd5-812b-2aae06b8fbcc\") " Aug 04 18:46:30.921855 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.921826 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4" (OuterVolumeSpecName: "kube-api-access-v25w4") pod "66a301ff-638b-4cd5-812b-2aae06b8fbcc" (UID: "66a301ff-638b-4cd5-812b-2aae06b8fbcc"). InnerVolumeSpecName "kube-api-access-v25w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:30.995805 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.995760 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:30.996246 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.996231 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" containerName="test-curl-7546-140211071667488" Aug 04 18:46:30.996289 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.996248 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" containerName="test-curl-7546-140211071667488" Aug 04 18:46:30.996335 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.996324 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" containerName="test-curl-7546-140211071667488" Aug 04 18:46:30.999639 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:30.999617 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:31.005430 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.005396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:31.020185 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.020138 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v25w4\" (UniqueName: \"kubernetes.io/projected/66a301ff-638b-4cd5-812b-2aae06b8fbcc-kube-api-access-v25w4\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:31.120919 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.120874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrn6l\" (UniqueName: \"kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l\") pod \"test-curl-7546-140211071521104\" (UID: \"2f26f3d0-ba01-44d0-8f10-3c09334181ee\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:31.222302 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.222194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrn6l\" (UniqueName: \"kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l\") pod \"test-curl-7546-140211071521104\" (UID: \"2f26f3d0-ba01-44d0-8f10-3c09334181ee\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:31.232492 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.232448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrn6l\" (UniqueName: \"kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l\") pod \"test-curl-7546-140211071521104\" (UID: \"2f26f3d0-ba01-44d0-8f10-3c09334181ee\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:31.311844 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.311787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:31.460097 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.460068 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:31.460896 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:31.460876 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f26f3d0_ba01_44d0_8f10_3c09334181ee.slice/crio-c5b7ab774f072efb6b8c6d5fb0f18804374218288155f3aec7f1ef81d3a56aff WatchSource:0}: Error finding container c5b7ab774f072efb6b8c6d5fb0f18804374218288155f3aec7f1ef81d3a56aff: Status 404 returned error can't find the container with id c5b7ab774f072efb6b8c6d5fb0f18804374218288155f3aec7f1ef81d3a56aff Aug 04 18:46:31.659952 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.659917 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:31.660141 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.659958 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9ad69d427da920a864baaa86fb36dc25f4ff82ae4f52e7e0aaa160094a6b14" Aug 04 18:46:31.665233 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:31.665189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071521104" event={"ID":"2f26f3d0-ba01-44d0-8f10-3c09334181ee","Type":"ContainerStarted","Data":"c5b7ab774f072efb6b8c6d5fb0f18804374218288155f3aec7f1ef81d3a56aff"} Aug 04 18:46:31.672410 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:31.672378 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071667488\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:32.673160 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:32.673128 2559 generic.go:356] "Generic (PLEG): container finished" podID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" containerID="290e645e2f6b44a3b58bf6ea41cc7807ca86c8db5c181625e17470c7b29ee0e1" exitCode=0 Aug 04 18:46:32.673564 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:32.673209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071521104" event={"ID":"2f26f3d0-ba01-44d0-8f10-3c09334181ee","Type":"ContainerDied","Data":"290e645e2f6b44a3b58bf6ea41cc7807ca86c8db5c181625e17470c7b29ee0e1"} Aug 04 18:46:32.675629 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:32.675595 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071667488\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" pod="opendatahub/test-curl-7546-140211071667488" Aug 04 18:46:32.739459 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:32.739428 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a301ff-638b-4cd5-812b-2aae06b8fbcc" path="/var/lib/kubelet/pods/66a301ff-638b-4cd5-812b-2aae06b8fbcc/volumes" Aug 04 18:46:33.813867 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.813838 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:33.857232 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.857201 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071521104_2f26f3d0-ba01-44d0-8f10-3c09334181ee/test-curl-7546-140211071521104/0.log" Aug 04 18:46:33.895427 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.895396 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:33.897600 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.897582 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:33.948271 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.948200 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrn6l\" (UniqueName: \"kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l\") pod \"2f26f3d0-ba01-44d0-8f10-3c09334181ee\" (UID: \"2f26f3d0-ba01-44d0-8f10-3c09334181ee\") " Aug 04 18:46:33.950408 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:33.950386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l" (OuterVolumeSpecName: "kube-api-access-rrn6l") pod "2f26f3d0-ba01-44d0-8f10-3c09334181ee" (UID: "2f26f3d0-ba01-44d0-8f10-3c09334181ee"). InnerVolumeSpecName "kube-api-access-rrn6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:34.025877 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.025844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:34.026249 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.026237 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" containerName="test-curl-7546-140211071521104" Aug 04 18:46:34.026295 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.026251 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" containerName="test-curl-7546-140211071521104" Aug 04 18:46:34.026330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.026324 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" containerName="test-curl-7546-140211071521104" Aug 04 18:46:34.029302 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.029288 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.049085 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.049054 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:34.049440 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.049418 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrn6l\" (UniqueName: \"kubernetes.io/projected/2f26f3d0-ba01-44d0-8f10-3c09334181ee-kube-api-access-rrn6l\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:34.150316 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.150286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9zd\" (UniqueName: \"kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd\") pod \"test-curl-7546-140211071521104\" (UID: \"df16370f-699f-4d7e-8982-c5637c723678\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.251251 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.251165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9zd\" (UniqueName: \"kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd\") pod \"test-curl-7546-140211071521104\" (UID: \"df16370f-699f-4d7e-8982-c5637c723678\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.262242 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.262222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9zd\" (UniqueName: \"kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd\") pod \"test-curl-7546-140211071521104\" (UID: \"df16370f-699f-4d7e-8982-c5637c723678\") " pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.337783 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.337753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.481526 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.481478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:34.483445 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:46:34.483422 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf16370f_699f_4d7e_8982_c5637c723678.slice/crio-d2133c4baf75ab0470f37bbdada7993b6e72e42abe560cb166e46b11544cce12 WatchSource:0}: Error finding container d2133c4baf75ab0470f37bbdada7993b6e72e42abe560cb166e46b11544cce12: Status 404 returned error can't find the container with id d2133c4baf75ab0470f37bbdada7993b6e72e42abe560cb166e46b11544cce12 Aug 04 18:46:34.686296 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.686256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071521104" event={"ID":"df16370f-699f-4d7e-8982-c5637c723678","Type":"ContainerStarted","Data":"d2133c4baf75ab0470f37bbdada7993b6e72e42abe560cb166e46b11544cce12"} Aug 04 18:46:34.687538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.687491 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5b7ab774f072efb6b8c6d5fb0f18804374218288155f3aec7f1ef81d3a56aff" Aug 04 18:46:34.687538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.687539 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:34.699133 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.699107 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7546-140211071521104" oldPodUID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" podUID="df16370f-699f-4d7e-8982-c5637c723678" Aug 04 18:46:34.739836 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:34.739806 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f26f3d0-ba01-44d0-8f10-3c09334181ee" path="/var/lib/kubelet/pods/2f26f3d0-ba01-44d0-8f10-3c09334181ee/volumes" Aug 04 18:46:35.694692 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:35.694656 2559 generic.go:356] "Generic (PLEG): container finished" podID="df16370f-699f-4d7e-8982-c5637c723678" containerID="5298b5f34edd3ed7bd74de82f36129cac55d92ac9c8819eca3b6a85734f7dd1f" exitCode=0 Aug 04 18:46:35.695079 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:35.694724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-140211071521104" event={"ID":"df16370f-699f-4d7e-8982-c5637c723678","Type":"ContainerDied","Data":"5298b5f34edd3ed7bd74de82f36129cac55d92ac9c8819eca3b6a85734f7dd1f"} Aug 04 18:46:36.842077 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.842051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:36.868901 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.868873 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7546-140211071521104_df16370f-699f-4d7e-8982-c5637c723678/test-curl-7546-140211071521104/0.log" Aug 04 18:46:36.894279 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.894247 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:36.898766 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.898741 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-140211071521104"] Aug 04 18:46:36.977612 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.977533 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp9zd\" (UniqueName: \"kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd\") pod \"df16370f-699f-4d7e-8982-c5637c723678\" (UID: \"df16370f-699f-4d7e-8982-c5637c723678\") " Aug 04 18:46:36.979697 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:36.979673 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd" (OuterVolumeSpecName: "kube-api-access-rp9zd") pod "df16370f-699f-4d7e-8982-c5637c723678" (UID: "df16370f-699f-4d7e-8982-c5637c723678"). InnerVolumeSpecName "kube-api-access-rp9zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:37.079104 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:37.079065 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9zd\" (UniqueName: \"kubernetes.io/projected/df16370f-699f-4d7e-8982-c5637c723678-kube-api-access-rp9zd\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:46:37.707748 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:37.707722 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2133c4baf75ab0470f37bbdada7993b6e72e42abe560cb166e46b11544cce12" Aug 04 18:46:37.707901 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:37.707755 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:37.717169 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:37.717137 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071521104\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="df16370f-699f-4d7e-8982-c5637c723678" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:38.739676 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:38.739641 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df16370f-699f-4d7e-8982-c5637c723678" path="/var/lib/kubelet/pods/df16370f-699f-4d7e-8982-c5637c723678/volumes" Aug 04 18:46:38.745698 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:46:38.745662 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7546-140211071521104\" is forbidden: User \"system:node:ip-10-0-128-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-51.ec2.internal' and this object" podUID="df16370f-699f-4d7e-8982-c5637c723678" pod="opendatahub/test-curl-7546-140211071521104" Aug 04 18:46:42.335202 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:42.335147 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-51.ec2.internal" podUID="c82be4bfc3f07367942d8a0fb59c52b4" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 18:46:51.786613 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:51.786576 2559 generic.go:356] "Generic (PLEG): container finished" podID="da150961-b2b1-4f17-a599-19507ad602ee" containerID="6ac96efdbac2cead50f82f074148e4f07de2dd5349022ca1aaab6327716130db" exitCode=1 Aug 04 18:46:51.786913 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:51.786656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" event={"ID":"da150961-b2b1-4f17-a599-19507ad602ee","Type":"ContainerDied","Data":"6ac96efdbac2cead50f82f074148e4f07de2dd5349022ca1aaab6327716130db"} Aug 04 18:46:51.786991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:51.786978 2559 scope.go:117] "RemoveContainer" containerID="6ac96efdbac2cead50f82f074148e4f07de2dd5349022ca1aaab6327716130db" Aug 04 18:46:52.640428 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.640396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 18:46:52.777135 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.777101 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" podUID="aef5e2ac-f46a-40f0-af21-3fd798a00236" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.44:8081/readyz\": dial tcp 10.133.0.44:8081: connect: connection refused" Aug 04 18:46:52.793365 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.793337 2559 generic.go:356] "Generic (PLEG): container finished" podID="4529511b-ad3f-408c-9344-b47586246793" containerID="82e727091b6b35499798747f47f52fff783b1fc924598f0b9838510069385c3c" exitCode=1 Aug 04 18:46:52.793707 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.793415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" event={"ID":"4529511b-ad3f-408c-9344-b47586246793","Type":"ContainerDied","Data":"82e727091b6b35499798747f47f52fff783b1fc924598f0b9838510069385c3c"} Aug 04 18:46:52.793767 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.793752 2559 scope.go:117] "RemoveContainer" containerID="82e727091b6b35499798747f47f52fff783b1fc924598f0b9838510069385c3c" Aug 04 18:46:52.794983 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.794967 2559 generic.go:356] "Generic (PLEG): container finished" podID="9fae65b2-3848-43fb-bf33-b9756d8e3d08" containerID="9a79b928de78629a1443cf16177ed9236017d8a55a0e831e853773aa7fced932" exitCode=1 Aug 04 18:46:52.795081 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.795043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" event={"ID":"9fae65b2-3848-43fb-bf33-b9756d8e3d08","Type":"ContainerDied","Data":"9a79b928de78629a1443cf16177ed9236017d8a55a0e831e853773aa7fced932"} Aug 04 18:46:52.795300 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.795287 2559 scope.go:117] "RemoveContainer" containerID="9a79b928de78629a1443cf16177ed9236017d8a55a0e831e853773aa7fced932" Aug 04 18:46:52.796934 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.796813 2559 generic.go:356] "Generic (PLEG): container finished" podID="537c67f7-4d6b-439a-a1f6-0fa79f4fc42d" containerID="f0a59a65754beb648e8df9719b38f9cb297c87b513af0c2f1a311217d35cefbf" exitCode=0 Aug 04 18:46:52.796934 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.796845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" event={"ID":"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d","Type":"ContainerDied","Data":"f0a59a65754beb648e8df9719b38f9cb297c87b513af0c2f1a311217d35cefbf"} Aug 04 18:46:52.797244 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.797221 2559 scope.go:117] "RemoveContainer" containerID="f0a59a65754beb648e8df9719b38f9cb297c87b513af0c2f1a311217d35cefbf" Aug 04 18:46:52.798653 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.798534 2559 generic.go:356] "Generic (PLEG): container finished" podID="aef5e2ac-f46a-40f0-af21-3fd798a00236" containerID="099dbbdfbe05f35763895d13775ee4e3c86c6a19e31b3b9777c491a354832db2" exitCode=1 Aug 04 18:46:52.798653 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.798602 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" event={"ID":"aef5e2ac-f46a-40f0-af21-3fd798a00236","Type":"ContainerDied","Data":"099dbbdfbe05f35763895d13775ee4e3c86c6a19e31b3b9777c491a354832db2"} Aug 04 18:46:52.798936 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.798906 2559 scope.go:117] "RemoveContainer" containerID="099dbbdfbe05f35763895d13775ee4e3c86c6a19e31b3b9777c491a354832db2" Aug 04 18:46:52.801003 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.800985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" event={"ID":"da150961-b2b1-4f17-a599-19507ad602ee","Type":"ContainerStarted","Data":"15a394f0ba0259cc188ff29ceab389ab378692dd47922c92e532f2943ee43df6"} Aug 04 18:46:52.801332 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:52.801315 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 18:46:53.311727 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.311693 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 18:46:53.812287 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.812245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" event={"ID":"537c67f7-4d6b-439a-a1f6-0fa79f4fc42d","Type":"ContainerStarted","Data":"79f0818cf62869b30338e88618a415ddd5a8e7d750f770fc6b14e75e401ff51f"} Aug 04 18:46:53.812818 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.812540 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:46:53.814309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.814275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" event={"ID":"aef5e2ac-f46a-40f0-af21-3fd798a00236","Type":"ContainerStarted","Data":"4217c67000cc885fe89785bf895f506612bae7f2746f5abf8ede460f055f4177"} Aug 04 18:46:53.814531 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.814480 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:46:53.816058 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.816034 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" event={"ID":"4529511b-ad3f-408c-9344-b47586246793","Type":"ContainerStarted","Data":"0f01d0c2c7490b19c85a01501b8c6cbaa6eda7ca56b5c4a33a57ce59ea46e6b9"} Aug 04 18:46:53.816183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.816100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 18:46:53.817808 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.817784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" event={"ID":"9fae65b2-3848-43fb-bf33-b9756d8e3d08","Type":"ContainerStarted","Data":"3adf09155548e27db3016f9146f4d439c94b6ad039aad7d5f48000461c568630"} Aug 04 18:46:53.818053 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:46:53.818035 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 18:47:03.819775 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:03.819742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-lfqwt" Aug 04 18:47:04.825993 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:04.825956 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-wz4z9" Aug 04 18:47:04.826410 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:04.826164 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pldx4" Aug 04 18:47:04.826410 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:04.826227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-2bx7m" Aug 04 18:47:24.826298 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:24.826261 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4r8n8" Aug 04 18:47:32.040534 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:32.040479 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:47:54.120693 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.120656 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:47:54.121095 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.121081 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df16370f-699f-4d7e-8982-c5637c723678" containerName="test-curl-7546-140211071521104" Aug 04 18:47:54.121139 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.121097 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df16370f-699f-4d7e-8982-c5637c723678" containerName="test-curl-7546-140211071521104" Aug 04 18:47:54.121174 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.121169 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df16370f-699f-4d7e-8982-c5637c723678" containerName="test-curl-7546-140211071521104" Aug 04 18:47:54.124351 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.124326 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.125700 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.125676 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsf8q\" (UniqueName: \"kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.125833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.125761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.125833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.125789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.128626 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.128603 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vrzn8\"" Aug 04 18:47:54.128737 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.128687 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:47:54.128962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.128946 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-071def-serving-cert\"" Aug 04 18:47:54.129037 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.129010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 18:47:54.142643 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.142617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:47:54.226493 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.226462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsf8q\" (UniqueName: \"kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.226714 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.226542 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.226714 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.226565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.226714 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:47:54.226681 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-serving-cert: secret "maas-api-e2e-ipp-b-071def-serving-cert" not found Aug 04 18:47:54.226852 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:47:54.226743 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls podName:0a720f6f-afbc-41a6-9bd0-41d24d6521e5 nodeName:}" failed. No retries permitted until 2026-08-04 18:47:54.72672612 +0000 UTC m=+3336.468880157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls") pod "maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" (UID: "0a720f6f-afbc-41a6-9bd0-41d24d6521e5") : secret "maas-api-e2e-ipp-b-071def-serving-cert" not found Aug 04 18:47:54.227126 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.227110 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.239996 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.239975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsf8q\" (UniqueName: \"kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.731818 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.731785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.734660 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.734634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.735361 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.735338 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:54.891910 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:54.891887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:47:54.893331 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:47:54.893308 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a720f6f_afbc_41a6_9bd0_41d24d6521e5.slice/crio-5cca73e0204e3bf009c4e3983d73daf0232a256a6e3ea091db4da764263c49cd WatchSource:0}: Error finding container 5cca73e0204e3bf009c4e3983d73daf0232a256a6e3ea091db4da764263c49cd: Status 404 returned error can't find the container with id 5cca73e0204e3bf009c4e3983d73daf0232a256a6e3ea091db4da764263c49cd Aug 04 18:47:55.179451 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:55.179417 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" event={"ID":"0a720f6f-afbc-41a6-9bd0-41d24d6521e5","Type":"ContainerStarted","Data":"5cca73e0204e3bf009c4e3983d73daf0232a256a6e3ea091db4da764263c49cd"} Aug 04 18:47:57.127541 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:57.127487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:47:57.193891 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:57.193854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" event={"ID":"0a720f6f-afbc-41a6-9bd0-41d24d6521e5","Type":"ContainerStarted","Data":"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c"} Aug 04 18:47:57.194049 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:57.193918 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:47:57.241334 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:47:57.241281 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" podStartSLOduration=1.132952779 podStartE2EDuration="3.241264432s" totalImagesPullingTime="2.108311653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:47:57.239454334 +0000 UTC m=+3338.981608392" watchObservedRunningTime="2026-08-04 18:47:57.241264432 +0000 UTC m=+3338.983418490" Aug 04 18:48:03.205351 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:48:03.205322 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:48:36.688888 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:48:36.688850 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:49:02.003238 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:02.003204 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:49:13.933854 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:13.933816 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:49:13.937326 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:13.937304 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:13.940038 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:13.940019 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f2f60a-openshift-default-dockercfg-2zkzh\"" Aug 04 18:49:13.954447 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:13.954424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:49:14.077485 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077485 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9px6\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.077742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.077722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178382 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178382 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178425 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9px6\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178890 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178682 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178890 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.178890 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.179053 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.179053 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.178942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.179161 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.179109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.179218 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.179179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.179278 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.179234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.180859 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.180842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.181220 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.181204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.188048 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.187993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.189001 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.188981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9px6\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6\") pod \"e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.247759 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.247725 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:14.400810 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.400778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:49:14.401843 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.401802 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:14.401943 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.401886 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:14.401943 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.401925 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:14.651881 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.651847 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerStarted","Data":"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a"} Aug 04 18:49:14.652077 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.651912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerStarted","Data":"cf5ecea44800115af0231bcc963c277a9b06bc08499b4a90b7558dbbdfba5ada"} Aug 04 18:49:14.680656 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:14.680601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podStartSLOduration=1.680576147 podStartE2EDuration="1.680576147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:14.679738921 +0000 UTC m=+3416.421892992" watchObservedRunningTime="2026-08-04 18:49:14.680576147 +0000 UTC m=+3416.422730204" Aug 04 18:49:15.248357 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:15.248321 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:15.253039 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:15.253017 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:15.657988 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:15.657954 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:15.659013 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:15.658992 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:20.811365 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.811331 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:20.815000 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.814983 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:20.823525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.823477 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f2f60a-serving-cert\"" Aug 04 18:49:20.831805 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.831782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:20.947365 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.947323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:20.947597 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.947389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:20.947597 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:20.947553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.048309 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.048276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.048468 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.048340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.048468 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.048416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.048593 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:49:21.048570 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-serving-cert: secret "maas-api-e2e-ipp-cleanup-f2f60a-serving-cert" not found Aug 04 18:49:21.048664 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:49:21.048653 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls podName:6172f75e-1242-4e02-a388-cf50b637e26a nodeName:}" failed. No retries permitted until 2026-08-04 18:49:21.548633956 +0000 UTC m=+3423.290787999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" (UID: "6172f75e-1242-4e02-a388-cf50b637e26a") : secret "maas-api-e2e-ipp-cleanup-f2f60a-serving-cert" not found Aug 04 18:49:21.049016 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.048996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.062391 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.062332 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.553674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.553644 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.556337 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.556308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.725438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.725407 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:21.880903 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:21.880864 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:21.885248 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:49:21.885220 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6172f75e_1242_4e02_a388_cf50b637e26a.slice/crio-1a63e73ec4a9a8478e764b6f945dc2a55639d4df188195724be7ac5b533157cf WatchSource:0}: Error finding container 1a63e73ec4a9a8478e764b6f945dc2a55639d4df188195724be7ac5b533157cf: Status 404 returned error can't find the container with id 1a63e73ec4a9a8478e764b6f945dc2a55639d4df188195724be7ac5b533157cf Aug 04 18:49:22.700487 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:22.700447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" event={"ID":"6172f75e-1242-4e02-a388-cf50b637e26a","Type":"ContainerStarted","Data":"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb"} Aug 04 18:49:22.700690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:22.700529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" event={"ID":"6172f75e-1242-4e02-a388-cf50b637e26a","Type":"ContainerStarted","Data":"1a63e73ec4a9a8478e764b6f945dc2a55639d4df188195724be7ac5b533157cf"} Aug 04 18:49:22.700690 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:22.700565 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:22.722895 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:22.722839 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" podStartSLOduration=2.128745408 podStartE2EDuration="2.72282045s" totalImagesPullingTime="594.075042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:22.720869349 +0000 UTC m=+3424.463023407" watchObservedRunningTime="2026-08-04 18:49:22.72282045 +0000 UTC m=+3424.464974510" Aug 04 18:49:24.485737 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:24.485704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:49:28.713317 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:28.713289 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:34.793616 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.793584 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerID="1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a" exitCode=0 Aug 04 18:49:34.794034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.793657 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerDied","Data":"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a"} Aug 04 18:49:34.794084 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.794047 2559 scope.go:117] "RemoveContainer" containerID="1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a" Aug 04 18:49:34.795286 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.795257 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:34.795417 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.795342 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:34.795417 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:34.795388 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Aug 04 18:49:35.802455 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:35.802424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerStarted","Data":"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa"} Aug 04 18:49:36.248442 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:36.248333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:36.248713 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:36.248686 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Aug 04 18:49:36.248791 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:36.248764 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Aug 04 18:49:37.248264 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:37.248222 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Aug 04 18:49:37.248775 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:37.248293 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Aug 04 18:49:38.248680 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:38.248643 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Aug 04 18:49:38.249070 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:38.248699 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Aug 04 18:49:39.248962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:39.248921 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Aug 04 18:49:39.249364 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:39.248980 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Aug 04 18:49:40.249098 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:40.249063 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Aug 04 18:49:40.249479 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:40.249130 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Aug 04 18:49:41.252990 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:41.252959 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:41.253363 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:41.253231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:41.253919 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:41.253900 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:49:49.286683 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.286651 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:49.287080 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.286886 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" podUID="6172f75e-1242-4e02-a388-cf50b637e26a" containerName="maas-api" containerID="cri-o://77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb" gracePeriod=30 Aug 04 18:49:49.529191 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.529167 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:49.630716 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.630687 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk\") pod \"6172f75e-1242-4e02-a388-cf50b637e26a\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " Aug 04 18:49:49.630930 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.630759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca\") pod \"6172f75e-1242-4e02-a388-cf50b637e26a\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " Aug 04 18:49:49.630930 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.630855 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") pod \"6172f75e-1242-4e02-a388-cf50b637e26a\" (UID: \"6172f75e-1242-4e02-a388-cf50b637e26a\") " Aug 04 18:49:49.631182 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.631159 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6172f75e-1242-4e02-a388-cf50b637e26a" (UID: "6172f75e-1242-4e02-a388-cf50b637e26a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:49:49.631267 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.631237 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6172f75e-1242-4e02-a388-cf50b637e26a-openshift-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:49:49.633222 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.633188 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk" (OuterVolumeSpecName: "kube-api-access-cxrtk") pod "6172f75e-1242-4e02-a388-cf50b637e26a" (UID: "6172f75e-1242-4e02-a388-cf50b637e26a"). InnerVolumeSpecName "kube-api-access-cxrtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:49:49.633398 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.633374 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6172f75e-1242-4e02-a388-cf50b637e26a" (UID: "6172f75e-1242-4e02-a388-cf50b637e26a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:49:49.732724 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.732641 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6172f75e-1242-4e02-a388-cf50b637e26a-maas-api-tls\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:49:49.732724 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.732668 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrtk\" (UniqueName: \"kubernetes.io/projected/6172f75e-1242-4e02-a388-cf50b637e26a-kube-api-access-cxrtk\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:49:49.889075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.889036 2559 generic.go:356] "Generic (PLEG): container finished" podID="6172f75e-1242-4e02-a388-cf50b637e26a" containerID="77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb" exitCode=0 Aug 04 18:49:49.889269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.889084 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" Aug 04 18:49:49.889269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.889084 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" event={"ID":"6172f75e-1242-4e02-a388-cf50b637e26a","Type":"ContainerDied","Data":"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb"} Aug 04 18:49:49.889269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.889215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6" event={"ID":"6172f75e-1242-4e02-a388-cf50b637e26a","Type":"ContainerDied","Data":"1a63e73ec4a9a8478e764b6f945dc2a55639d4df188195724be7ac5b533157cf"} Aug 04 18:49:49.889269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.889248 2559 scope.go:117] "RemoveContainer" containerID="77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb" Aug 04 18:49:49.899262 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.899238 2559 scope.go:117] "RemoveContainer" containerID="77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb" Aug 04 18:49:49.899541 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:49:49.899517 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb\": container with ID starting with 77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb not found: ID does not exist" containerID="77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb" Aug 04 18:49:49.899614 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.899551 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb"} err="failed to get container status \"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb\": rpc error: code = NotFound desc = could not find container \"77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb\": container with ID starting with 77fd9f9d8985378ebc5eac662cc822fc12b57b240924577ae53a0c32448238cb not found: ID does not exist" Aug 04 18:49:49.919708 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.919683 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:49.922045 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:49.922026 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f2f60a-595d466cdc-kdzq6"] Aug 04 18:49:50.740195 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:50.740163 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6172f75e-1242-4e02-a388-cf50b637e26a" path="/var/lib/kubelet/pods/6172f75e-1242-4e02-a388-cf50b637e26a/volumes" Aug 04 18:49:52.299097 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:52.299057 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:49:57.351083 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:57.351050 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:49:57.351531 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:49:57.351249 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" containerID="cri-o://b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa" gracePeriod=30 Aug 04 18:50:01.374072 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.374037 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw"] Aug 04 18:50:01.374457 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.374424 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172f75e-1242-4e02-a388-cf50b637e26a" containerName="maas-api" Aug 04 18:50:01.374457 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.374434 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172f75e-1242-4e02-a388-cf50b637e26a" containerName="maas-api" Aug 04 18:50:01.374570 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.374518 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6172f75e-1242-4e02-a388-cf50b637e26a" containerName="maas-api" Aug 04 18:50:01.376422 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.376399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:01.379563 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.379522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:50:01.389569 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.389544 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw"] Aug 04 18:50:01.553274 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.553238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw\" (UID: \"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:01.654428 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.654340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw\" (UID: \"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:01.655053 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.655032 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw\" (UID: \"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:01.686211 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.686171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:01.822175 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.822142 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw"] Aug 04 18:50:01.822798 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:50:01.822768 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2d6480_3ffe_4c4c_b638_4d7a1ea1936c.slice/crio-29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385 WatchSource:0}: Error finding container 29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385: Status 404 returned error can't find the container with id 29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385 Aug 04 18:50:01.963807 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:01.963729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" event={"ID":"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c","Type":"ContainerStarted","Data":"29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385"} Aug 04 18:50:02.584452 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.584425 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:50:02.764771 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764741 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.764927 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.764927 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.764927 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764893 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764969 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9px6\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.764991 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765024 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765075 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765071 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs\") pod \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\" (UID: \"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930\") " Aug 04 18:50:02.765317 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765109 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:02.765317 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765264 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:02.765440 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765315 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:02.765440 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765351 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:50:02.765572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765478 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.765572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765522 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.765572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765540 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.765572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765554 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.765850 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.765827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:02.767102 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.767076 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:50:02.767269 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.767252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:02.767335 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.767252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:50:02.767464 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.767448 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6" (OuterVolumeSpecName: "kube-api-access-v9px6") pod "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" (UID: "5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930"). InnerVolumeSpecName "kube-api-access-v9px6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:50:02.866730 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.866695 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9px6\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-kube-api-access-v9px6\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.866730 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.866728 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.866730 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.866738 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.866961 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.866747 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.866961 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.866754 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:02.972611 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.972482 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerID="b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa" exitCode=0 Aug 04 18:50:02.972611 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.972556 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" Aug 04 18:50:02.972611 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.972555 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerDied","Data":"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa"} Aug 04 18:50:02.972911 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.972629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm" event={"ID":"5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930","Type":"ContainerDied","Data":"cf5ecea44800115af0231bcc963c277a9b06bc08499b4a90b7558dbbdfba5ada"} Aug 04 18:50:02.972911 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.972660 2559 scope.go:117] "RemoveContainer" containerID="b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa" Aug 04 18:50:02.974572 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.974542 2559 generic.go:356] "Generic (PLEG): container finished" podID="6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" containerID="996a55ce0e1da9226df3c1e1e0e930dd96799cfc57d46d2e9b137001abf39b61" exitCode=0 Aug 04 18:50:02.974701 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.974613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" event={"ID":"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c","Type":"ContainerDied","Data":"996a55ce0e1da9226df3c1e1e0e930dd96799cfc57d46d2e9b137001abf39b61"} Aug 04 18:50:02.983029 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.983012 2559 scope.go:117] "RemoveContainer" containerID="1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a" Aug 04 18:50:02.994525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.993034 2559 scope.go:117] "RemoveContainer" containerID="b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa" Aug 04 18:50:02.994525 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:50:02.993379 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa\": container with ID starting with b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa not found: ID does not exist" containerID="b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa" Aug 04 18:50:02.994525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.993410 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa"} err="failed to get container status \"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa\": rpc error: code = NotFound desc = could not find container \"b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa\": container with ID starting with b6be3a89bf920b7dfba540d8c45d6e1e7224e68da297e8967449cdf062dcc0fa not found: ID does not exist" Aug 04 18:50:02.994525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.993435 2559 scope.go:117] "RemoveContainer" containerID="1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a" Aug 04 18:50:02.994525 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:50:02.994474 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a\": container with ID starting with 1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a not found: ID does not exist" containerID="1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a" Aug 04 18:50:02.994738 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:02.994524 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a"} err="failed to get container status \"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a\": rpc error: code = NotFound desc = could not find container \"1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a\": container with ID starting with 1d1552761b9bdb6898ec52ec1a618063d568b75c97e4fc6a7ee012d49d37a65a not found: ID does not exist" Aug 04 18:50:03.020416 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:03.020386 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:50:03.027282 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:03.027258 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f2f60a-openshift-default-865dcb97bb-ppdrm"] Aug 04 18:50:04.111721 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.111697 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:04.279970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.279884 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca\") pod \"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c\" (UID: \"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c\") " Aug 04 18:50:04.280217 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.280195 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" (UID: "6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:50:04.380991 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.380955 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c-maas-api-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:50:04.740033 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.740001 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" path="/var/lib/kubelet/pods/5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930/volumes" Aug 04 18:50:04.988538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.988486 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" Aug 04 18:50:04.988538 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.988529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw" event={"ID":"6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c","Type":"ContainerDied","Data":"29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385"} Aug 04 18:50:04.988795 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:04.988577 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a19dfc2474d336eb64a12d3a04b836d66ff0fc732e3282ab7945b4869a2385" Aug 04 18:50:14.742530 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:14.742469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:50:20.768740 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:20.768701 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw"] Aug 04 18:50:20.771741 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:20.771714 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6a6d9f-l2mgw"] Aug 04 18:50:22.744819 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:22.740567 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" path="/var/lib/kubelet/pods/6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c/volumes" Aug 04 18:50:23.904819 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:50:23.904779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:51:19.849438 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:19.849404 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:51:19.850069 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:19.849641 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" podUID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" containerName="maas-api" containerID="cri-o://11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c" gracePeriod=30 Aug 04 18:51:20.105945 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.105865 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:51:20.174823 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.174789 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca\") pod \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " Aug 04 18:51:20.175012 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.174858 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") pod \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " Aug 04 18:51:20.175012 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.174906 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsf8q\" (UniqueName: \"kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q\") pod \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\" (UID: \"0a720f6f-afbc-41a6-9bd0-41d24d6521e5\") " Aug 04 18:51:20.175219 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.175194 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a720f6f-afbc-41a6-9bd0-41d24d6521e5" (UID: "0a720f6f-afbc-41a6-9bd0-41d24d6521e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:20.177136 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.177104 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a720f6f-afbc-41a6-9bd0-41d24d6521e5" (UID: "0a720f6f-afbc-41a6-9bd0-41d24d6521e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:51:20.177136 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.177103 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q" (OuterVolumeSpecName: "kube-api-access-hsf8q") pod "0a720f6f-afbc-41a6-9bd0-41d24d6521e5" (UID: "0a720f6f-afbc-41a6-9bd0-41d24d6521e5"). InnerVolumeSpecName "kube-api-access-hsf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:20.276719 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.276685 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-openshift-service-ca\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:51:20.276719 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.276714 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-maas-api-tls\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:51:20.276719 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.276724 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsf8q\" (UniqueName: \"kubernetes.io/projected/0a720f6f-afbc-41a6-9bd0-41d24d6521e5-kube-api-access-hsf8q\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:51:20.439139 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.439044 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" containerID="11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c" exitCode=0 Aug 04 18:51:20.439139 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.439097 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" Aug 04 18:51:20.439139 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.439124 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" event={"ID":"0a720f6f-afbc-41a6-9bd0-41d24d6521e5","Type":"ContainerDied","Data":"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c"} Aug 04 18:51:20.439380 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.439178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd" event={"ID":"0a720f6f-afbc-41a6-9bd0-41d24d6521e5","Type":"ContainerDied","Data":"5cca73e0204e3bf009c4e3983d73daf0232a256a6e3ea091db4da764263c49cd"} Aug 04 18:51:20.439380 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.439201 2559 scope.go:117] "RemoveContainer" containerID="11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c" Aug 04 18:51:20.448732 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.448716 2559 scope.go:117] "RemoveContainer" containerID="11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c" Aug 04 18:51:20.449013 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:51:20.448995 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c\": container with ID starting with 11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c not found: ID does not exist" containerID="11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c" Aug 04 18:51:20.449066 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.449021 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c"} err="failed to get container status \"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c\": rpc error: code = NotFound desc = could not find container \"11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c\": container with ID starting with 11f1a410a15800ab60870b188e364e527e256ecb0738708d013a8f890c47f64c not found: ID does not exist" Aug 04 18:51:20.479443 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.479414 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:51:20.481551 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.481529 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-071def-78d45f7f77-7lzfd"] Aug 04 18:51:20.739870 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:20.739795 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" path="/var/lib/kubelet/pods/0a720f6f-afbc-41a6-9bd0-41d24d6521e5/volumes" Aug 04 18:51:22.092384 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:22.092353 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:51:37.390768 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:37.390734 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:51:50.383423 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:50.383389 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-m727d"] Aug 04 18:51:55.599886 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:55.599856 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:51:55.600381 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:55.600064 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" containerID="cri-o://7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee" gracePeriod=30 Aug 04 18:51:56.943876 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:56.943832 2559 patch_prober.go:28] interesting pod/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:51:56.944266 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:51:56.943898 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:52:00.843671 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.843646 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:52:00.968409 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968409 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968409 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968385 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968412 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968448 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js752\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968485 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968561 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968687 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968702 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:00.968797 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968770 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert\") pod \"687504cf-df66-4d4b-950f-1bbabacbc214\" (UID: \"687504cf-df66-4d4b-950f-1bbabacbc214\") " Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968915 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data" (OuterVolumeSpecName: "istio-data") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.968949 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.969130 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.969152 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-credential-socket\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.969166 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-data\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:00.969172 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.969176 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-workload-certs\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:00.969730 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.969177 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:52:00.970783 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.970762 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:52:00.970888 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.970867 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token" (OuterVolumeSpecName: "istio-token") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:52:00.971070 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.971050 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:52:00.971199 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:00.971182 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752" (OuterVolumeSpecName: "kube-api-access-js752") pod "687504cf-df66-4d4b-950f-1bbabacbc214" (UID: "687504cf-df66-4d4b-950f-1bbabacbc214"). InnerVolumeSpecName "kube-api-access-js752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:52:01.069932 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.069898 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687504cf-df66-4d4b-950f-1bbabacbc214-istio-envoy\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:01.069932 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.069930 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687504cf-df66-4d4b-950f-1bbabacbc214-istio-podinfo\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:01.069932 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.069938 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js752\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-kube-api-access-js752\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:01.070157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.069947 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687504cf-df66-4d4b-950f-1bbabacbc214-istio-token\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:01.070157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.069958 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687504cf-df66-4d4b-950f-1bbabacbc214-istiod-ca-cert\") on node \"ip-10-0-128-51.ec2.internal\" DevicePath \"\"" Aug 04 18:52:01.697121 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.697090 2559 generic.go:356] "Generic (PLEG): container finished" podID="687504cf-df66-4d4b-950f-1bbabacbc214" containerID="7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee" exitCode=0 Aug 04 18:52:01.697306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.697140 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" Aug 04 18:52:01.697306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.697183 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerDied","Data":"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee"} Aug 04 18:52:01.697306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.697237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z" event={"ID":"687504cf-df66-4d4b-950f-1bbabacbc214","Type":"ContainerDied","Data":"2b23e3287c42a90d5d54f15442e16bfa1dd29f7933df4f000b8c9b3c99848b91"} Aug 04 18:52:01.697306 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.697260 2559 scope.go:117] "RemoveContainer" containerID="7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee" Aug 04 18:52:01.707217 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.707202 2559 scope.go:117] "RemoveContainer" containerID="eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b" Aug 04 18:52:01.715947 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.715756 2559 scope.go:117] "RemoveContainer" containerID="7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee" Aug 04 18:52:01.716014 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:52:01.716001 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee\": container with ID starting with 7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee not found: ID does not exist" containerID="7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee" Aug 04 18:52:01.716056 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.716023 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee"} err="failed to get container status \"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee\": rpc error: code = NotFound desc = could not find container \"7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee\": container with ID starting with 7fae1d99851dc176b146d64a66014cea8cc37dae12638735fdb1377075bfaeee not found: ID does not exist" Aug 04 18:52:01.716056 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.716040 2559 scope.go:117] "RemoveContainer" containerID="eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b" Aug 04 18:52:01.716235 ip-10-0-128-51 kubenswrapper[2559]: E0804 18:52:01.716219 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b\": container with ID starting with eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b not found: ID does not exist" containerID="eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b" Aug 04 18:52:01.716276 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.716238 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b"} err="failed to get container status \"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b\": rpc error: code = NotFound desc = could not find container \"eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b\": container with ID starting with eb0177482b8cb974d8d3e657e621bc8a573eaf31bec41d40d25f6f57f5bb143b not found: ID does not exist" Aug 04 18:52:01.725859 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.725837 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:52:01.728144 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:01.728124 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bc18f1-openshift-default-6cc9cdcf76-lfv2z"] Aug 04 18:52:02.157764 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.157735 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4r8n8_4529511b-ad3f-408c-9344-b47586246793/manager/1.log" Aug 04 18:52:02.405661 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.405636 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-l666w_2ca8fc11-9ec3-4bef-825d-d1ce99693438/server/0.log" Aug 04 18:52:02.534228 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.534203 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pldx4_9fae65b2-3848-43fb-bf33-b9756d8e3d08/manager/1.log" Aug 04 18:52:02.739379 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.739335 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" path="/var/lib/kubelet/pods/687504cf-df66-4d4b-950f-1bbabacbc214/volumes" Aug 04 18:52:02.910817 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.910735 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r_6a4a2a8d-30ab-4c39-9a0f-47ca8c800709/manager/0.log" Aug 04 18:52:02.917359 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:02.917332 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r_6a4a2a8d-30ab-4c39-9a0f-47ca8c800709/copy-manifests/0.log" Aug 04 18:52:04.758915 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:04.758887 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-lfqwt_da150961-b2b1-4f17-a599-19507ad602ee/manager/1.log" Aug 04 18:52:04.883251 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:04.883222 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-52rkg_9fe634df-a104-4a5e-b98f-b5118c091669/manager/0.log" Aug 04 18:52:05.000831 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:05.000803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-lkr26_4bcb83f6-63ff-443f-a0b1-8a0dbf15622d/kuadrant-console-plugin/0.log" Aug 04 18:52:05.129264 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:05.129232 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-wz4z9_537c67f7-4d6b-439a-a1f6-0fa79f4fc42d/manager/1.log" Aug 04 18:52:05.249431 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:05.249401 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-m727d_44454a20-5faf-44f8-a208-8a486716e174/limitador/0.log" Aug 04 18:52:05.368539 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:05.368487 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-2bx7m_aef5e2ac-f46a-40f0-af21-3fd798a00236/manager/1.log" Aug 04 18:52:05.741898 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:05.741874 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7_d8180508-a3f5-48d5-a440-788eef175f41/istio-proxy/0.log" Aug 04 18:52:06.116081 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:06.116056 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-jhv6z_79f2c87a-30c0-4652-a9a5-80baf5933464/kube-auth-proxy/0.log" Aug 04 18:52:06.244685 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:06.244653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/1.log" Aug 04 18:52:07.370426 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:07.370391 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw7mtw_046dee85-87a1-4004-ae27-847eaa3e5269/main/0.log" Aug 04 18:52:07.490272 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:07.490239 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-nttk9_805bc751-44a8-41f3-b9eb-f9b97a8a8dba/main/0.log" Aug 04 18:52:13.597345 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597263 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v4pcl/must-gather-d6fdb"] Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597639 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" containerName="maas-api" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597649 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" containerName="maas-api" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597664 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597670 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597686 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" containerName="revoke-keys" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597691 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" containerName="revoke-keys" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597701 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597706 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597712 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597717 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597723 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.597758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597728 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597777 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2d6480-3ffe-4c4c-b638-4d7a1ea1936c" containerName="revoke-keys" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597784 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597790 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a720f6f-afbc-41a6-9bd0-41d24d6521e5" containerName="maas-api" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597796 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597803 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="687504cf-df66-4d4b-950f-1bbabacbc214" containerName="istio-proxy" Aug 04 18:52:13.598125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.597810 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9d8b9e-2d7d-4c4f-bc85-735d19a3c930" containerName="istio-proxy" Aug 04 18:52:13.600480 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.600462 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.603321 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.603297 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4pcl\"/\"openshift-service-ca.crt\"" Aug 04 18:52:13.603452 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.603337 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4pcl\"/\"kube-root-ca.crt\"" Aug 04 18:52:13.604045 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.604030 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v4pcl\"/\"default-dockercfg-hmk57\"" Aug 04 18:52:13.611026 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.611007 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/must-gather-d6fdb"] Aug 04 18:52:13.684822 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.684791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8c26\" (UniqueName: \"kubernetes.io/projected/5ba381ba-c67b-49b4-96c4-4d0582de4290-kube-api-access-v8c26\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.684992 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.684830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ba381ba-c67b-49b4-96c4-4d0582de4290-must-gather-output\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.785588 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.785557 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8c26\" (UniqueName: \"kubernetes.io/projected/5ba381ba-c67b-49b4-96c4-4d0582de4290-kube-api-access-v8c26\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.785758 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.785599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ba381ba-c67b-49b4-96c4-4d0582de4290-must-gather-output\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.785967 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.785950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ba381ba-c67b-49b4-96c4-4d0582de4290-must-gather-output\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.794659 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.794637 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8c26\" (UniqueName: \"kubernetes.io/projected/5ba381ba-c67b-49b4-96c4-4d0582de4290-kube-api-access-v8c26\") pod \"must-gather-d6fdb\" (UID: \"5ba381ba-c67b-49b4-96c4-4d0582de4290\") " pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:13.911102 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:13.911025 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" Aug 04 18:52:14.051728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:14.051693 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/must-gather-d6fdb"] Aug 04 18:52:14.053008 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:52:14.052982 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ba381ba_c67b_49b4_96c4_4d0582de4290.slice/crio-9502eda288aab74fe8759656e036157dd10a637ae914b3e6686f28ae12fa7251 WatchSource:0}: Error finding container 9502eda288aab74fe8759656e036157dd10a637ae914b3e6686f28ae12fa7251: Status 404 returned error can't find the container with id 9502eda288aab74fe8759656e036157dd10a637ae914b3e6686f28ae12fa7251 Aug 04 18:52:14.054900 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:14.054883 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:52:14.777941 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:14.777902 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" event={"ID":"5ba381ba-c67b-49b4-96c4-4d0582de4290","Type":"ContainerStarted","Data":"9502eda288aab74fe8759656e036157dd10a637ae914b3e6686f28ae12fa7251"} Aug 04 18:52:15.789047 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:15.789003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" event={"ID":"5ba381ba-c67b-49b4-96c4-4d0582de4290","Type":"ContainerStarted","Data":"121687a4d106013c94482ab0933ce351de336d2633850fe8e9ee2f3b0c2dfe6a"} Aug 04 18:52:15.789428 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:15.789058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" event={"ID":"5ba381ba-c67b-49b4-96c4-4d0582de4290","Type":"ContainerStarted","Data":"a5a4ef3cf4b7c415b47b70015b364ba544ae0a2a171a2e465d31b59628182839"} Aug 04 18:52:15.809739 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:15.809683 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v4pcl/must-gather-d6fdb" podStartSLOduration=2.152937572 podStartE2EDuration="2.809665615s" totalImagesPullingTime="656.728043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:15.807257271 +0000 UTC m=+3597.549411332" watchObservedRunningTime="2026-08-04 18:52:15.809665615 +0000 UTC m=+3597.551819674" Aug 04 18:52:16.414555 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:16.414311 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fwwnn_2875059c-8a6a-424b-ba19-66e66511f8de/global-pull-secret-syncer/0.log" Aug 04 18:52:16.745825 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:16.745749 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xbr79_ac47e548-6eab-4a87-af8f-89dbe85ee4f4/konnectivity-agent/0.log" Aug 04 18:52:16.830305 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:16.830273 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-51.ec2.internal_c82be4bfc3f07367942d8a0fb59c52b4/haproxy/0.log" Aug 04 18:52:21.448598 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.447363 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-lfqwt_da150961-b2b1-4f17-a599-19507ad602ee/manager/1.log" Aug 04 18:52:21.449562 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.449534 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-lfqwt_da150961-b2b1-4f17-a599-19507ad602ee/manager/0.log" Aug 04 18:52:21.478987 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.478953 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-52rkg_9fe634df-a104-4a5e-b98f-b5118c091669/manager/0.log" Aug 04 18:52:21.510559 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.510531 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-lkr26_4bcb83f6-63ff-443f-a0b1-8a0dbf15622d/kuadrant-console-plugin/0.log" Aug 04 18:52:21.571604 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.571576 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-wz4z9_537c67f7-4d6b-439a-a1f6-0fa79f4fc42d/manager/1.log" Aug 04 18:52:21.621061 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.620992 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-wz4z9_537c67f7-4d6b-439a-a1f6-0fa79f4fc42d/manager/0.log" Aug 04 18:52:21.657055 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.657025 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-m727d_44454a20-5faf-44f8-a208-8a486716e174/limitador/0.log" Aug 04 18:52:21.706891 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.706815 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-2bx7m_aef5e2ac-f46a-40f0-af21-3fd798a00236/manager/1.log" Aug 04 18:52:21.760105 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:21.760073 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-2bx7m_aef5e2ac-f46a-40f0-af21-3fd798a00236/manager/0.log" Aug 04 18:52:23.461918 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:23.461891 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-pqvwh_faf2b139-3acf-4b43-a573-df0e930845fa/kube-state-metrics/0.log" Aug 04 18:52:23.495267 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:23.495237 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-pqvwh_faf2b139-3acf-4b43-a573-df0e930845fa/kube-rbac-proxy-main/0.log" Aug 04 18:52:23.546377 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:23.546344 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-pqvwh_faf2b139-3acf-4b43-a573-df0e930845fa/kube-rbac-proxy-self/0.log" Aug 04 18:52:23.622525 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:23.621876 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6dff57d5f6-7q2nx_14dbb70e-ee82-484e-b7fa-5d47b6f57ca6/metrics-server/0.log" Aug 04 18:52:24.110468 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.110437 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbsxj_12b99295-bad7-40f7-b3ae-359ae84592a4/node-exporter/0.log" Aug 04 18:52:24.147589 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.147567 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbsxj_12b99295-bad7-40f7-b3ae-359ae84592a4/kube-rbac-proxy/0.log" Aug 04 18:52:24.199215 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.199184 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jbsxj_12b99295-bad7-40f7-b3ae-359ae84592a4/init-textfile/0.log" Aug 04 18:52:24.314742 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.314714 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-52pm5_5a223ae7-ebbe-4ab4-a002-7a3606c5e301/kube-rbac-proxy-main/0.log" Aug 04 18:52:24.371463 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.371346 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-52pm5_5a223ae7-ebbe-4ab4-a002-7a3606c5e301/kube-rbac-proxy-self/0.log" Aug 04 18:52:24.427728 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.427700 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-52pm5_5a223ae7-ebbe-4ab4-a002-7a3606c5e301/openshift-state-metrics/0.log" Aug 04 18:52:24.787403 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.787374 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-mkmzp_30a155f1-d79e-43cc-8f08-7b1655785630/prometheus-operator-admission-webhook/0.log" Aug 04 18:52:24.822954 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.822923 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f77f59b44-ldgvv_3bbee3bc-6ba1-4162-a850-b7d93684e62f/telemeter-client/0.log" Aug 04 18:52:24.855537 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.855491 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f77f59b44-ldgvv_3bbee3bc-6ba1-4162-a850-b7d93684e62f/reload/0.log" Aug 04 18:52:24.900811 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:24.900784 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f77f59b44-ldgvv_3bbee3bc-6ba1-4162-a850-b7d93684e62f/kube-rbac-proxy/0.log" Aug 04 18:52:26.413889 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.413849 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv"] Aug 04 18:52:26.420034 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.420000 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430157 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-lib-modules\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnwk\" (UniqueName: \"kubernetes.io/projected/851b1eca-b7c7-41f5-8ebd-f802962d3f51-kube-api-access-zwnwk\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430330 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-proc\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430445 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-podres\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430530 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-sys\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.430876 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.430852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv"] Aug 04 18:52:26.531726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-lib-modules\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.531726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnwk\" (UniqueName: \"kubernetes.io/projected/851b1eca-b7c7-41f5-8ebd-f802962d3f51-kube-api-access-zwnwk\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.531726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-proc\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.531726 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-podres\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.531970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-sys\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.531970 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531851 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-sys\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.532073 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.531978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-lib-modules\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.533905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.532326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-proc\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.533905 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.532430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/851b1eca-b7c7-41f5-8ebd-f802962d3f51-podres\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.544185 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.544160 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnwk\" (UniqueName: \"kubernetes.io/projected/851b1eca-b7c7-41f5-8ebd-f802962d3f51-kube-api-access-zwnwk\") pod \"perf-node-gather-daemonset-vk2rv\" (UID: \"851b1eca-b7c7-41f5-8ebd-f802962d3f51\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.740594 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.740387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:26.949183 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:26.948564 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv"] Aug 04 18:52:26.960962 ip-10-0-128-51 kubenswrapper[2559]: W0804 18:52:26.960911 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod851b1eca_b7c7_41f5_8ebd_f802962d3f51.slice/crio-41c029dc0abd22a5a27e822f8718d735660d337fc51373682808889f035b5b0a WatchSource:0}: Error finding container 41c029dc0abd22a5a27e822f8718d735660d337fc51373682808889f035b5b0a: Status 404 returned error can't find the container with id 41c029dc0abd22a5a27e822f8718d735660d337fc51373682808889f035b5b0a Aug 04 18:52:27.640651 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:27.640625 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64b94997cc-bzrqb_36be4884-2377-4d61-89e8-bc428dc49d01/console/0.log" Aug 04 18:52:27.901981 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:27.901879 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" event={"ID":"851b1eca-b7c7-41f5-8ebd-f802962d3f51","Type":"ContainerStarted","Data":"3c10c8a9dd0ab101c87c5ff46988a9a17ec94295da195cb5a1766331a5ed8e03"} Aug 04 18:52:27.901981 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:27.901952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" event={"ID":"851b1eca-b7c7-41f5-8ebd-f802962d3f51","Type":"ContainerStarted","Data":"41c029dc0abd22a5a27e822f8718d735660d337fc51373682808889f035b5b0a"} Aug 04 18:52:27.902833 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:27.902804 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:27.926565 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:27.926514 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" podStartSLOduration=1.926479547 podStartE2EDuration="1.926479547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:27.924759004 +0000 UTC m=+3609.666913064" watchObservedRunningTime="2026-08-04 18:52:27.926479547 +0000 UTC m=+3609.668633607" Aug 04 18:52:29.515602 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:29.515573 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-722jp_b955947d-6c9c-423d-9ac3-f6eeb9449497/dns/0.log" Aug 04 18:52:29.543624 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:29.543594 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-722jp_b955947d-6c9c-423d-9ac3-f6eeb9449497/kube-rbac-proxy/0.log" Aug 04 18:52:29.746199 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:29.745927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mm72d_ff2c6d80-d62d-4ec6-9e08-1524b1d75a5c/dns-node-resolver/0.log" Aug 04 18:52:30.323698 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:30.323664 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c97cb6f48-s6xxc_3895845c-8fc8-4876-9f26-001964bfdca1/registry/0.log" Aug 04 18:52:30.349577 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:30.349544 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-66lz4_e6805167-3bd3-4ebd-8f5f-7af6d2537f56/node-ca/0.log" Aug 04 18:52:31.416211 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:31.416182 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7_d8180508-a3f5-48d5-a440-788eef175f41/istio-proxy/0.log" Aug 04 18:52:31.665125 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:31.665099 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-jhv6z_79f2c87a-30c0-4652-a9a5-80baf5933464/kube-auth-proxy/0.log" Aug 04 18:52:31.704973 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:31.704891 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/0.log" Aug 04 18:52:31.713115 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:31.713088 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/1.log" Aug 04 18:52:32.738488 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:32.738460 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ttb2_5efcd9cf-4967-4fba-8dd4-6cc4b2d83a94/serve-healthcheck-canary/0.log" Aug 04 18:52:33.417647 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:33.417615 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d4klp_911a9056-9587-41aa-b68e-2a2bd3a7a21e/kube-rbac-proxy/0.log" Aug 04 18:52:33.447254 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:33.447226 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d4klp_911a9056-9587-41aa-b68e-2a2bd3a7a21e/exporter/0.log" Aug 04 18:52:33.474141 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:33.474116 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d4klp_911a9056-9587-41aa-b68e-2a2bd3a7a21e/extractor/0.log" Aug 04 18:52:34.940077 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:34.940049 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-vk2rv" Aug 04 18:52:35.948072 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:35.947794 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4r8n8_4529511b-ad3f-408c-9344-b47586246793/manager/1.log" Aug 04 18:52:36.177348 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.177318 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4r8n8_4529511b-ad3f-408c-9344-b47586246793/manager/0.log" Aug 04 18:52:36.367890 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.367858 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-l666w_2ca8fc11-9ec3-4bef-825d-d1ce99693438/server/0.log" Aug 04 18:52:36.425097 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.425067 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pldx4_9fae65b2-3848-43fb-bf33-b9756d8e3d08/manager/1.log" Aug 04 18:52:36.471210 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.471174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pldx4_9fae65b2-3848-43fb-bf33-b9756d8e3d08/manager/0.log" Aug 04 18:52:36.718291 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.718213 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r_6a4a2a8d-30ab-4c39-9a0f-47ca8c800709/manager/0.log" Aug 04 18:52:36.748674 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:36.748587 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-qmw7r_6a4a2a8d-30ab-4c39-9a0f-47ca8c800709/copy-manifests/0.log" Aug 04 18:52:37.216962 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:37.216925 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7_d8180508-a3f5-48d5-a440-788eef175f41/istio-proxy/0.log" Aug 04 18:52:37.412854 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:37.412818 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-jhv6z_79f2c87a-30c0-4652-a9a5-80baf5933464/kube-auth-proxy/0.log" Aug 04 18:52:37.437652 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:37.437621 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/0.log" Aug 04 18:52:37.447249 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:37.447112 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/1.log" Aug 04 18:52:38.174811 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:38.174782 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-v8dbx_4e2e8c1d-cd51-4b61-befc-f22433a0c5ac/openshift-lws-operator/0.log" Aug 04 18:52:38.995702 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:38.995623 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-6v4h7_d8180508-a3f5-48d5-a440-788eef175f41/istio-proxy/0.log" Aug 04 18:52:39.197777 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:39.197746 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-jhv6z_79f2c87a-30c0-4652-a9a5-80baf5933464/kube-auth-proxy/0.log" Aug 04 18:52:39.220019 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:39.219992 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/0.log" Aug 04 18:52:39.230484 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:39.230460 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-zq5d6_577f2833-3643-484a-9fd9-95cf35f72f5f/istio-proxy/1.log" Aug 04 18:52:44.616608 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.616571 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8vbnl_fe50a6e9-b81d-4e3e-81e9-942ecde2bbca/kube-multus/0.log" Aug 04 18:52:44.837466 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.837438 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/kube-multus-additional-cni-plugins/0.log" Aug 04 18:52:44.861096 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.861072 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/egress-router-binary-copy/0.log" Aug 04 18:52:44.883693 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.883627 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/cni-plugins/0.log" Aug 04 18:52:44.906466 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.906443 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/bond-cni-plugin/0.log" Aug 04 18:52:44.928254 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.928224 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/routeoverride-cni/0.log" Aug 04 18:52:44.956050 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.956030 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/whereabouts-cni-bincopy/0.log" Aug 04 18:52:44.979693 ip-10-0-128-51 kubenswrapper[2559]: I0804 18:52:44.979665 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-p6jwf_e99de878-d603-4183-8a3c-4bec85213382/whereabouts-cni/0.log"