Jul 23 18:04:25.362742 ip-10-0-132-197 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 18:04:25.362873 ip-10-0-132-197 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 18:04:25.362887 ip-10-0-132-197 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 18:04:25.363140 ip-10-0-132-197 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 18:04:35.365175 ip-10-0-132-197 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 18:04:35.365197 ip-10-0-132-197 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot be10100cca1044209600d29e6f4951f8 -- Jul 23 18:06:34.691882 ip-10-0-132-197 systemd[1]: Starting Kubernetes Kubelet... Jul 23 18:06:36.866975 ip-10-0-132-197 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:06:36.866975 ip-10-0-132-197 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 18:06:36.866975 ip-10-0-132-197 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:06:36.866975 ip-10-0-132-197 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 18:06:36.866975 ip-10-0-132-197 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 18:06:36.868854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.868765 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 18:06:36.873869 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873849 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.873869 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873869 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.873869 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873872 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873877 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873881 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873884 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873887 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873890 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873893 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873896 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873899 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873901 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873912 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873915 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873918 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873921 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873924 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873927 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873929 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873932 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873934 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873937 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.873983 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873940 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873942 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873945 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873948 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873952 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873956 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873960 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873962 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873965 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873968 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873971 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873974 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873977 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873980 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873984 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873988 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873990 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873993 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.873996 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.874464 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874000 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874004 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874007 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874009 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874012 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874015 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874017 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874020 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874023 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874025 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874028 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874030 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874033 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874036 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874038 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874042 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874045 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874047 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874050 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.874919 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874054 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874056 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874059 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874062 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874064 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874068 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874071 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874076 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874079 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874082 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874086 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874089 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874092 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874095 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874098 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874101 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874103 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874106 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874108 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.875383 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874112 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874115 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874117 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874120 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874122 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874125 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874128 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874131 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874133 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874136 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874139 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874142 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874145 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874148 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874151 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874154 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874156 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874159 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874162 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874164 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.875827 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874897 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874904 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874907 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874910 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874914 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874917 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874920 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874922 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874925 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874928 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874931 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874934 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874937 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874939 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874942 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874944 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874947 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874949 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874952 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874954 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.876333 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874957 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874960 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874962 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874965 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874968 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874971 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874974 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874977 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874979 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874982 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874985 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874988 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874991 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874993 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874996 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.874998 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875001 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875004 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875007 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875010 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.876889 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875012 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875015 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875017 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875020 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875022 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875025 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875027 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875030 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875032 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875035 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875037 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875040 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875042 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875045 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875048 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875050 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875053 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875056 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875059 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875061 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.877377 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875064 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875066 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875069 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875073 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875077 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875080 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875083 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875085 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875088 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875091 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875094 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875096 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875099 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875101 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875105 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875109 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875112 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875115 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875120 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.877872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875124 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875127 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875130 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875132 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875135 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875138 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875141 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875144 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875146 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875149 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875152 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875155 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875157 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875160 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875163 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875165 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875168 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875171 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875173 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875177 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.878322 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875561 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875565 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875568 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875572 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875574 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875577 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875580 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875582 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875585 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875587 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875590 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875593 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875595 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875598 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875600 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875603 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875607 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875609 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875612 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875615 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.878816 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875618 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875620 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875623 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875626 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875628 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875631 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875633 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875636 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875639 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875642 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875645 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875648 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875651 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875654 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875657 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875659 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875662 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875665 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875667 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.879287 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875670 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875672 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875675 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875677 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875680 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875682 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875685 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875692 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875694 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875697 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875699 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875702 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875705 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875707 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875710 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875714 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875718 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875721 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875723 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.879727 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875726 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875732 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875735 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875738 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875754 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875758 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875761 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875764 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875767 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875770 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875773 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875776 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875779 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875781 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875784 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875787 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875789 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875792 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875795 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.880188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875798 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875802 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875805 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875807 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875810 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875812 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875816 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875819 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875821 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875824 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875826 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875829 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875832 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875834 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875838 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875841 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875844 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875847 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875850 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875852 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.880634 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875854 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.875857 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875938 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875946 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875953 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875958 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875963 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875966 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875970 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875975 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875978 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875981 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875985 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875990 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875993 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875996 2570 flags.go:64] FLAG: --cgroup-root="" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.875998 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876001 2570 flags.go:64] FLAG: --client-ca-file="" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876004 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876007 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876011 2570 flags.go:64] FLAG: --cluster-domain="" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876014 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876017 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876021 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 18:06:36.881155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876024 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876028 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876033 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876036 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876039 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876042 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876045 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876048 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876051 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876054 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876059 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876062 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876065 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876068 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876071 2570 flags.go:64] FLAG: --enable-server="true" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876074 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876553 2570 flags.go:64] FLAG: --event-burst="100" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876557 2570 flags.go:64] FLAG: --event-qps="50" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876560 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876564 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876567 2570 flags.go:64] FLAG: --eviction-hard="" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876572 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876575 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876578 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876581 2570 flags.go:64] FLAG: --eviction-soft="" Jul 23 18:06:36.881771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876583 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876586 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876589 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876592 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876595 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876598 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876601 2570 flags.go:64] FLAG: --feature-gates="" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876605 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876608 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876615 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876618 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876621 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876624 2570 flags.go:64] FLAG: --help="false" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876627 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-132-197.ec2.internal" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876630 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876633 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876636 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876639 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876643 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876646 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876648 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876651 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876659 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 18:06:36.882395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876662 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876665 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876668 2570 flags.go:64] FLAG: --kube-reserved="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876671 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876674 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876677 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876680 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876683 2570 flags.go:64] FLAG: --lock-file="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876685 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876689 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876691 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876701 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876704 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876707 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876710 2570 flags.go:64] FLAG: --logging-format="text" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876712 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876716 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876719 2570 flags.go:64] FLAG: --manifest-url="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876723 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876727 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876730 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876735 2570 flags.go:64] FLAG: --max-pods="110" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876737 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876760 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876763 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 18:06:36.882993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876766 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876770 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876773 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876776 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876784 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876786 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876790 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876793 2570 flags.go:64] FLAG: --pod-cidr="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876796 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876802 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876805 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876808 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876811 2570 flags.go:64] FLAG: --port="10250" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876814 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876818 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c8abfbee0def6640" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876821 2570 flags.go:64] FLAG: --qos-reserved="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876824 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876827 2570 flags.go:64] FLAG: --register-node="true" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876831 2570 flags.go:64] FLAG: --register-with-taints="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876834 2570 flags.go:64] FLAG: --registry-burst="10" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876837 2570 flags.go:64] FLAG: --registry-qps="5" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876840 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876843 2570 flags.go:64] FLAG: --reserved-memory="" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876846 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876850 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 18:06:36.883600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876854 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876857 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876860 2570 flags.go:64] FLAG: --runonce="false" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876863 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876866 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876869 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876872 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876875 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876878 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876881 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876885 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876888 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876891 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876894 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876897 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876900 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876904 2570 flags.go:64] FLAG: --system-cgroups="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876907 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876912 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876915 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876918 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876922 2570 flags.go:64] FLAG: --tls-min-version="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876925 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876928 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876931 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 18:06:36.884428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876933 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876937 2570 flags.go:64] FLAG: --v="2" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876941 2570 flags.go:64] FLAG: --version="false" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876945 2570 flags.go:64] FLAG: --vmodule="" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876949 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.876952 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877088 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877093 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877098 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877101 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877104 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877107 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877110 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877113 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877116 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877121 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877124 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877127 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877130 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877134 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877136 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.885560 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877139 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877142 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877146 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877149 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877151 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877154 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877156 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877159 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877162 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877164 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877167 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877171 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877174 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877178 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877181 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877185 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877188 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877192 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877194 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.886380 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877197 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877200 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877203 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877205 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877208 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877210 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877213 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877217 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877220 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877223 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877225 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877228 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877230 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877233 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877235 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877238 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877241 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877244 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877253 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877257 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.886961 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877259 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877262 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877265 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877268 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877271 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877273 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877276 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877279 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877282 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877284 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877287 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877289 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877292 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877294 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877297 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877299 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877303 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877305 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877308 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.887434 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877312 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877315 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877318 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877322 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877325 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877328 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877331 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877333 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877336 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877338 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877341 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877344 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877346 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877349 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877351 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877354 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877357 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877359 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877362 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.887898 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877365 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877367 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877370 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877372 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877375 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877377 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.877380 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.877963 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.887269 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.887290 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.888045 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.888053 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888118 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888125 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.888367 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888130 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888134 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888137 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888140 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888143 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888146 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888149 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888152 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888154 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888157 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888160 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888163 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888165 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888168 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888171 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888173 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888176 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888179 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888181 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888184 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.888713 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888187 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888189 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888192 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888195 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888197 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888201 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888206 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888209 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888212 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888216 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888218 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888221 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888224 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888226 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888229 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888231 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888234 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888237 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888239 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.889209 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888242 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888245 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888247 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888250 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888252 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888255 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888258 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888260 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888263 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888265 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888268 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888271 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888273 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888276 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888279 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888282 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888285 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888288 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888290 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888293 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.889681 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888297 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888300 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888302 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888305 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888308 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888312 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888315 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888318 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888320 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888323 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888326 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888328 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888331 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888333 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888336 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888339 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888341 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888344 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888348 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.890165 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888351 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888354 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888356 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888359 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888362 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888365 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888368 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888371 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888373 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888376 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888379 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888382 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888385 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888388 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888391 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888394 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888397 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888399 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.890631 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888402 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.888408 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888529 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888535 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888538 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888541 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888543 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888547 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888550 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888553 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888556 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888559 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888561 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888564 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888568 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 18:06:36.891093 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888571 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888574 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888576 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888579 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888582 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888584 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888587 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888591 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888595 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888598 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888602 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888605 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888607 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888610 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888613 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888615 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888618 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888620 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888623 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888625 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 18:06:36.891462 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888628 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888630 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888633 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888636 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888638 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888641 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888643 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888646 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888648 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888651 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888655 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888658 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888662 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888665 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888667 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888670 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888673 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888675 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888678 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888680 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 18:06:36.891959 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888683 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888686 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888689 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888692 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888694 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888697 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888700 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888702 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888705 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888708 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888710 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888713 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888716 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888719 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888721 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888724 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888727 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888730 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888732 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888735 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 18:06:36.892468 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888737 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888751 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888754 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888758 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888761 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888766 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888768 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888771 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888774 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888777 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888780 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888782 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888785 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888788 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888791 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888794 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888797 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888800 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888802 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 18:06:36.893014 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888805 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888808 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888811 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888814 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888817 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888819 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:36.888822 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.888828 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.888990 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 18:06:36.893474 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.892303 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 18:06:36.893721 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.893708 2570 server.go:1019] "Starting client certificate rotation" Jul 23 18:06:36.893870 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.893779 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 18:06:36.893870 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.893824 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 18:06:36.896766 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.896738 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 18:06:36.899408 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.899389 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 18:06:36.909360 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.909339 2570 log.go:25] "Validated CRI v1 runtime API" Jul 23 18:06:36.911650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.911628 2570 log.go:25] "Validated CRI v1 image API" Jul 23 18:06:36.912527 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.912513 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 18:06:36.917379 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.917360 2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d528a46d-87aa-45bb-ba44-072cab6bb5ed:/dev/nvme0n1p3 d6a7bd8f-21cc-483c-be22-00c5095f4250:/dev/nvme0n1p4] Jul 23 18:06:36.917429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.917379 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 18:06:36.923483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.923341 2570 manager.go:216] Machine: {Timestamp:2026-07-23 18:06:36.922607081 +0000 UTC m=+0.490204243 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107941 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d028574b2dd4b0ba797f1fd973fab SystemUUID:ec2d0285-74b2-dd4b-0ba7-97f1fd973fab BootID:be10100c-ca10-4420-9600-d29e6f4951f8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:19:a8:48:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:19:a8:48:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:d8:e1:8b:96:64 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 18:06:36.923483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.923467 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 18:06:36.923633 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.923556 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 18:06:36.926918 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.926885 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 18:06:36.927098 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.926923 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 18:06:36.928304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.928291 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 18:06:36.928379 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.928311 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 18:06:36.928379 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.928323 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 18:06:36.928983 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.928971 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 18:06:36.930180 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.930165 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 18:06:36.931685 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.931666 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 18:06:36.933400 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933385 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 23 18:06:36.933596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933584 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 18:06:36.933656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933626 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 23 18:06:36.933656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933641 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 18:06:36.933760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933657 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 18:06:36.933760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933681 2570 kubelet.go:398] "Adding apiserver pod source" Jul 23 18:06:36.933760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.933694 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 18:06:36.934101 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.934088 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 18:06:36.934171 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.934111 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 18:06:36.937088 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.937072 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 18:06:36.938475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.938461 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 18:06:36.938536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.938491 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 18:06:36.939227 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939216 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 18:06:36.939265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939232 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 18:06:36.939265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939238 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 18:06:36.939265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939243 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 18:06:36.939265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939253 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 18:06:36.939265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939260 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939268 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939274 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939281 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939287 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939297 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 18:06:36.939399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.939303 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 18:06:36.940547 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.940532 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 18:06:36.940580 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.940550 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 18:06:36.943756 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.943719 2570 server.go:1268] "Started kubelet" Jul 23 18:06:36.943847 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.943816 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 18:06:36.944041 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.943986 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 18:06:36.944145 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.944129 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 18:06:36.944686 ip-10-0-132-197 systemd[1]: Started Kubernetes Kubelet. Jul 23 18:06:36.945735 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.945721 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 23 18:06:36.947908 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.947891 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 18:06:36.948486 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.948454 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 18:06:36.951309 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.951290 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 18:06:36.951869 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.951849 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 18:06:36.951915 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.951848 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 18:06:36.952132 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.951258 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-197.ec2.internal.18c4fd6d004a3e91 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-197.ec2.internal,UID:ip-10-0-132-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-197.ec2.internal,},FirstTimestamp:2026-07-23 18:06:36.943687313 +0000 UTC m=+0.511284476,LastTimestamp:2026-07-23 18:06:36.943687313 +0000 UTC m=+0.511284476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-197.ec2.internal,}" Jul 23 18:06:36.953862 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.953845 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 18:06:36.953944 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.953851 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 18:06:36.954038 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.954018 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:36.954517 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954503 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 18:06:36.954637 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954505 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 18:06:36.954637 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954620 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 18:06:36.954827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954680 2570 factory.go:55] Registering systemd factory Jul 23 18:06:36.954827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954694 2570 factory.go:223] Registration of the systemd container factory successfully Jul 23 18:06:36.954827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954697 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 23 18:06:36.954827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.954712 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 23 18:06:36.955203 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955185 2570 factory.go:153] Registering CRI-O factory Jul 23 18:06:36.955283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955207 2570 factory.go:223] Registration of the crio container factory successfully Jul 23 18:06:36.955283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955261 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 18:06:36.955407 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955287 2570 factory.go:103] Registering Raw factory Jul 23 18:06:36.955407 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955301 2570 manager.go:1196] Started watching for new ooms in manager Jul 23 18:06:36.955898 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.955884 2570 manager.go:349] Starting recovery of all containers Jul 23 18:06:36.956367 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.956335 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 18:06:36.956451 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.956431 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 18:06:36.961627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.961602 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vgstz" Jul 23 18:06:36.967041 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.967018 2570 manager.go:354] Recovery completed Jul 23 18:06:36.968928 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:36.968812 2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 23 18:06:36.970018 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.969995 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vgstz" Jul 23 18:06:36.972130 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.972114 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:36.974047 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974032 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:36.974125 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974060 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:36.974125 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974071 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:36.974702 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974690 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 18:06:36.974702 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974701 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 18:06:36.974821 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974715 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 23 18:06:36.974821 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.974771 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:36.975443 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975426 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:36.975521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975458 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:36.975521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975474 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:36.975947 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975935 2570 policy_none.go:49] "None policy: Start" Jul 23 18:06:36.975998 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975951 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 18:06:36.975998 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.975962 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 18:06:36.976919 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:36.976910 2570 policy_none.go:47] "Start" Jul 23 18:06:37.014626 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.014611 2570 manager.go:337] "Starting Device Plugin manager" Jul 23 18:06:37.014717 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.014647 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 18:06:37.014717 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.014659 2570 server.go:87] "Starting device plugin registration server" Jul 23 18:06:37.014987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.014973 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 18:06:37.015044 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.014990 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 18:06:37.015095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.015078 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 18:06:37.015188 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.015164 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 18:06:37.015188 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.015173 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 18:06:37.015860 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.015833 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 18:06:37.015943 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.015896 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.115288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.115249 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:37.116482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.116345 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:37.116482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.116381 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:37.116482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.116394 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:37.116482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.116421 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.123406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.123356 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.123406 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.123381 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-197.ec2.internal\": node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.134914 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.134875 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 18:06:37.136165 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.136145 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 18:06:37.136275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.136184 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 18:06:37.136275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.136208 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 18:06:37.136275 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.136239 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 18:06:37.139325 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.139309 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:37.143524 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.143508 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.236800 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.236731 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal"] Jul 23 18:06:37.236998 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.236852 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:37.238431 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.238414 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:37.238517 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.238445 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:37.238517 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.238457 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:37.239675 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.239662 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:37.239781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.239768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.239847 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.239799 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:37.240443 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240426 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:37.240536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240456 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:37.240536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240470 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:37.240536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240482 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:37.240536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240506 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:37.240536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.240518 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:37.241592 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.241579 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.241637 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.241603 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 18:06:37.242266 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.242250 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientMemory" Jul 23 18:06:37.242345 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.242279 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 18:06:37.242345 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.242291 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeHasSufficientPID" Jul 23 18:06:37.243555 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.243541 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.255651 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.255618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.255795 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.255654 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.255795 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.255679 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd05e8e976bdb21bacf0ae5f21b68be6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-197.ec2.internal\" (UID: \"bd05e8e976bdb21bacf0ae5f21b68be6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.275700 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.275675 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-197.ec2.internal\" not found" node="ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.280269 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.280249 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-197.ec2.internal\" not found" node="ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.343789 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.343762 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.356263 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.356347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.356347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd05e8e976bdb21bacf0ae5f21b68be6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-197.ec2.internal\" (UID: \"bd05e8e976bdb21bacf0ae5f21b68be6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.356347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356330 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd05e8e976bdb21bacf0ae5f21b68be6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-197.ec2.internal\" (UID: \"bd05e8e976bdb21bacf0ae5f21b68be6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.356467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356348 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.356467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.356360 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fc05c03729111c4ef810e639cee95d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal\" (UID: \"1fc05c03729111c4ef810e639cee95d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.444658 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.444555 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.545078 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.545038 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.577551 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.577522 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.584429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.584401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.645392 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.645351 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.746051 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.745980 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.846661 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.846629 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-197.ec2.internal\" not found" Jul 23 18:06:37.857352 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.857312 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:37.875072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.875041 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:37.893180 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.893150 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 18:06:37.893332 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.893299 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:06:37.893395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.893321 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:06:37.893395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.893325 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 18:06:37.934524 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.934493 2570 apiserver.go:52] "Watching apiserver" Jul 23 18:06:37.954267 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.954237 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 18:06:37.954267 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.954237 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.954486 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.954266 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 18:06:37.955117 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.955094 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-5fs28","openshift-multus/multus-additional-cni-plugins-dd47g","openshift-multus/multus-s4mpv","kube-system/global-pull-secret-syncer-bmskp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7","openshift-cluster-node-tuning-operator/tuned-s5knm","openshift-image-registry/node-ca-wbsxq","openshift-multus/network-metrics-daemon-x2wcs","openshift-network-diagnostics/network-check-target-w79t6","openshift-network-operator/iptables-alerter-tgdq9","openshift-ovn-kubernetes/ovnkube-node-vwjpk","kube-system/konnectivity-agent-bx84d"] Jul 23 18:06:37.956584 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.956562 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:37.958049 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.958029 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:37.959120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.959102 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s4mpv" Jul 23 18:06:37.960174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960159 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:37.960258 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.960221 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:37.960369 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960308 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gvxct\"" Jul 23 18:06:37.960369 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.960452 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960383 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.960490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 18:06:37.960579 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960564 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.960622 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960575 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.960622 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.960576 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 18:06:37.961353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.961335 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 18:06:37.961453 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.961371 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:37.961755 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.961721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5985h\"" Jul 23 18:06:37.961826 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.961792 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 18:06:37.962518 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.962502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:37.962598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.962515 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qzh48\"" Jul 23 18:06:37.963757 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.963723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:37.964557 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.964538 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.964557 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.964533 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.964679 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.964575 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v96p8\"" Jul 23 18:06:37.964908 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.964891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:37.964994 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.964892 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 18:06:37.965338 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.965296 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:37.966060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.965455 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.966060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.965822 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.966060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.965822 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-96ddn\"" Jul 23 18:06:37.966293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.966249 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.966293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.966282 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 18:06:37.966409 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.966283 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vrmgl\"" Jul 23 18:06:37.966455 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.966394 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.968456 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.968434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:37.968544 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:37.968501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:37.969722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.969709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:37.969960 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.969912 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 18:06:37.970033 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.970022 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" Jul 23 18:06:37.971140 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.971123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:37.972521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.972442 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal"] Jul 23 18:06:37.972521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.972518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:37.972797 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.972782 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.973515 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973496 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 18:06:37.973580 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973496 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m9z8f\"" Jul 23 18:06:37.973669 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973630 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 18:01:36 +0000 UTC" deadline="2028-01-09 12:37:26.690872418 +0000 UTC" Jul 23 18:06:37.973703 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973676 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12834h30m48.717199353s" Jul 23 18:06:37.973703 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.974027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.973996 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 18:06:37.974411 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.974397 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 18:06:37.975610 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 18:06:37.975693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975643 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jxgdx\"" Jul 23 18:06:37.975828 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975813 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zszvg\"" Jul 23 18:06:37.975877 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975842 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 18:06:37.975918 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975885 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 18:06:37.975961 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.975951 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 18:06:37.976257 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.976245 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 18:06:37.978452 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.978435 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 18:06:37.978823 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.978810 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 18:06:37.984860 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.984842 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal"] Jul 23 18:06:37.984946 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.984862 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 18:06:37.994184 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:37.994163 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tr62x" Jul 23 18:06:38.004233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.004213 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tr62x" Jul 23 18:06:38.055684 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.055662 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 18:06:38.059096 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-host\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.059228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059102 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-kubelet-config\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.059228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059120 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-ovn\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059147 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059169 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cce2639-0478-49ca-a434-208bfc9dd501-ovn-node-metrics-cert\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-hosts-file\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-multus-certs\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059258 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49nr\" (UniqueName: \"kubernetes.io/projected/1c47f7ce-8505-4759-b65e-759b36cf1034-kube-api-access-f49nr\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-kubelet\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-script-lib\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-system-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059371 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-bin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059417 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-daemon-config\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059438 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-device-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-os-release\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-env-overrides\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-cnibin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-conf-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-etc-selinux\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059702 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-modprobe-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysconfig\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059787 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-netns\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.059857 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-systemd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9cpd\" (UniqueName: \"kubernetes.io/projected/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-kube-api-access-z9cpd\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-os-release\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059934 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-cni-binary-copy\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-multus\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.059994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11788ed9-1469-46aa-a70e-cb063de2b14a-host-slash\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060010 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-hostroot\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060077 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-host\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060113 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c8d624b-754c-45c4-98f6-848d0fc2f37b-konnectivity-ca\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-tmp-dir\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060200 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-netns\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-kubelet\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn49q\" (UniqueName: \"kubernetes.io/projected/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kube-api-access-cn49q\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-kubernetes\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060306 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfbb\" (UniqueName: \"kubernetes.io/projected/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-kube-api-access-sbfbb\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-dbus\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-log-socket\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060380 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-config\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-k8s-cni-cncf-io\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-serviceca\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nxf\" (UniqueName: \"kubernetes.io/projected/114cc0ef-9e50-4957-95ba-44398bf8de22-kube-api-access-v7nxf\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.060509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060505 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54klk\" (UniqueName: \"kubernetes.io/projected/9cce2639-0478-49ca-a434-208bfc9dd501-kube-api-access-54klk\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-etc-kubernetes\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nckp\" (UniqueName: \"kubernetes.io/projected/11788ed9-1469-46aa-a70e-cb063de2b14a-kube-api-access-6nckp\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-etc-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-bin\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-registration-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-conf\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-systemd\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060729 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-tuned\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060783 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-node-log\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060890 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-netd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-cnibin\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-socket-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.061046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-sys\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.060986 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-system-cni-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-var-lib-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061497 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c8d624b-754c-45c4-98f6-848d0fc2f37b-agent-certs\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hls\" (UniqueName: \"kubernetes.io/projected/ba60a11b-ba4f-4987-ac74-481c32323c2a-kube-api-access-t8hls\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11788ed9-1469-46aa-a70e-cb063de2b14a-iptables-alerter-script\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.061587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061582 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-systemd-units\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-tmp\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-slash\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.061776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-run\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-lib-modules\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.061951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-socket-dir-parent\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.061951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061839 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdqmk\" (UniqueName: \"kubernetes.io/projected/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-kube-api-access-xdqmk\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.061951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-sys-fs\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.064507 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.061888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-var-lib-kubelet\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.136988 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.136952 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05e8e976bdb21bacf0ae5f21b68be6.slice/crio-6751997e34d79d895728443d413cbfdf5425918c31961246f20a0b42206d4b24 WatchSource:0}: Error finding container 6751997e34d79d895728443d413cbfdf5425918c31961246f20a0b42206d4b24: Status 404 returned error can't find the container with id 6751997e34d79d895728443d413cbfdf5425918c31961246f20a0b42206d4b24 Jul 23 18:06:38.137729 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.137696 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc05c03729111c4ef810e639cee95d1.slice/crio-0858daa847b3d9075996765b719cbca048ca033bd0bd3241b930700b1127c4fc WatchSource:0}: Error finding container 0858daa847b3d9075996765b719cbca048ca033bd0bd3241b930700b1127c4fc: Status 404 returned error can't find the container with id 0858daa847b3d9075996765b719cbca048ca033bd0bd3241b930700b1127c4fc Jul 23 18:06:38.142924 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.142906 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:06:38.163278 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163278 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-kubelet\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-script-lib\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-system-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-system-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163380 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163380 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-kubelet\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-bin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-daemon-config\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163453 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-device-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-os-release\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163515 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-env-overrides\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-device-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-cnibin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-bin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-conf-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-etc-selinux\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-cnibin\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-modprobe-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysconfig\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.163647 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-os-release\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163684 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-netns\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-etc-selinux\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.163725 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:38.663701014 +0000 UTC m=+2.231298183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:38.163868 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163727 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysconfig\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-netns\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-systemd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-systemd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163795 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9cpd\" (UniqueName: \"kubernetes.io/projected/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-kube-api-access-z9cpd\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-os-release\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-modprobe-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-conf-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-cni-binary-copy\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-multus\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-os-release\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11788ed9-1469-46aa-a70e-cb063de2b14a-host-slash\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.163991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-hostroot\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-host\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c8d624b-754c-45c4-98f6-848d0fc2f37b-konnectivity-ca\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.164654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-env-overrides\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-tmp-dir\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164082 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-script-lib\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164082 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-daemon-config\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164124 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-netns\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11788ed9-1469-46aa-a70e-cb063de2b14a-host-slash\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-host\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164193 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-netns\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-kubelet\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164225 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-hostroot\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn49q\" (UniqueName: \"kubernetes.io/projected/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kube-api-access-cn49q\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-kubernetes\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfbb\" (UniqueName: \"kubernetes.io/projected/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-kube-api-access-sbfbb\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164311 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-dbus\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-log-socket\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.165489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164358 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-kubelet\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164363 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-config\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-k8s-cni-cncf-io\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164399 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-tmp-dir\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164404 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1c47f7ce-8505-4759-b65e-759b36cf1034-cni-binary-copy\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164402 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-d\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-log-socket\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164465 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-var-lib-cni-multus\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164477 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-k8s-cni-cncf-io\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164405 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-serviceca\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164500 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nxf\" (UniqueName: \"kubernetes.io/projected/114cc0ef-9e50-4957-95ba-44398bf8de22-kube-api-access-v7nxf\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164500 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-kubernetes\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54klk\" (UniqueName: \"kubernetes.io/projected/9cce2639-0478-49ca-a434-208bfc9dd501-kube-api-access-54klk\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-dbus\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-etc-kubernetes\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nckp\" (UniqueName: \"kubernetes.io/projected/11788ed9-1469-46aa-a70e-cb063de2b14a-kube-api-access-6nckp\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-etc-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-bin\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164649 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-registration-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164672 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-conf\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-etc-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164676 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-etc-kubernetes\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-systemd\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-tuned\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-bin\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164764 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-systemd\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-registration-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164806 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-serviceca\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164824 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-sysctl-conf\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-node-log\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-netd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.164968 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:38.166854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.164984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-cnibin\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.165013 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:38.664997725 +0000 UTC m=+2.232594880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cce2639-0478-49ca-a434-208bfc9dd501-ovnkube-config\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-cnibin\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-socket-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165052 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165076 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-node-log\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165096 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-sys\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-system-cni-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-socket-dir\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-var-lib-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-cni-netd\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c8d624b-754c-45c4-98f6-848d0fc2f37b-agent-certs\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hls\" (UniqueName: \"kubernetes.io/projected/ba60a11b-ba4f-4987-ac74-481c32323c2a-kube-api-access-t8hls\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165215 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-sys\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-var-lib-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11788ed9-1469-46aa-a70e-cb063de2b14a-iptables-alerter-script\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.167397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165288 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-systemd-units\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165295 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba60a11b-ba4f-4987-ac74-481c32323c2a-system-cni-dir\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165314 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba60a11b-ba4f-4987-ac74-481c32323c2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-tmp\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165412 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-systemd-units\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165443 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-slash\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-run\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-slash\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165590 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-lib-modules\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165615 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-socket-dir-parent\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165655 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-run\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdqmk\" (UniqueName: \"kubernetes.io/projected/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-kube-api-access-xdqmk\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-sys-fs\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.167949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-var-lib-kubelet\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165722 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-cni-dir\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165516 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c8d624b-754c-45c4-98f6-848d0fc2f37b-konnectivity-ca\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-host\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165795 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-host\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165797 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-kubelet-config\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165812 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11788ed9-1469-46aa-a70e-cb063de2b14a-iptables-alerter-script\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165838 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-multus-socket-dir-parent\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165842 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-kubelet-config\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-ovn\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-sys-fs\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cce2639-0478-49ca-a434-208bfc9dd501-ovn-node-metrics-cert\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165935 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-lib-modules\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165795 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-openvswitch\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-hosts-file\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.165994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-var-lib-kubelet\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.166008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-multus-certs\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.166033 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cce2639-0478-49ca-a434-208bfc9dd501-run-ovn\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.166034 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49nr\" (UniqueName: \"kubernetes.io/projected/1c47f7ce-8505-4759-b65e-759b36cf1034-kube-api-access-f49nr\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.166068 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1c47f7ce-8505-4759-b65e-759b36cf1034-host-run-multus-certs\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.166072 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-hosts-file\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.168347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-tmp\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.168385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-etc-tuned\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.168431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cce2639-0478-49ca-a434-208bfc9dd501-ovn-node-metrics-cert\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.168974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.168470 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c8d624b-754c-45c4-98f6-848d0fc2f37b-agent-certs\") pod \"konnectivity-agent-bx84d\" (UID: \"5c8d624b-754c-45c4-98f6-848d0fc2f37b\") " pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.177546 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.177498 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:38.177546 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.177523 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:38.177546 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.177536 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:38.177808 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.177607 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:38.677586802 +0000 UTC m=+2.245183971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:38.179529 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.179496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdqmk\" (UniqueName: \"kubernetes.io/projected/2e4b3d0d-f43f-46a1-a999-440f0985c9ac-kube-api-access-xdqmk\") pod \"node-ca-wbsxq\" (UID: \"2e4b3d0d-f43f-46a1-a999-440f0985c9ac\") " pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.179659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.179538 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hls\" (UniqueName: \"kubernetes.io/projected/ba60a11b-ba4f-4987-ac74-481c32323c2a-kube-api-access-t8hls\") pod \"multus-additional-cni-plugins-dd47g\" (UID: \"ba60a11b-ba4f-4987-ac74-481c32323c2a\") " pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.179659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.179560 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nxf\" (UniqueName: \"kubernetes.io/projected/114cc0ef-9e50-4957-95ba-44398bf8de22-kube-api-access-v7nxf\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.179819 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.179794 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54klk\" (UniqueName: \"kubernetes.io/projected/9cce2639-0478-49ca-a434-208bfc9dd501-kube-api-access-54klk\") pod \"ovnkube-node-vwjpk\" (UID: \"9cce2639-0478-49ca-a434-208bfc9dd501\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.179819 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.179809 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nckp\" (UniqueName: \"kubernetes.io/projected/11788ed9-1469-46aa-a70e-cb063de2b14a-kube-api-access-6nckp\") pod \"iptables-alerter-tgdq9\" (UID: \"11788ed9-1469-46aa-a70e-cb063de2b14a\") " pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.180169 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.180150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9cpd\" (UniqueName: \"kubernetes.io/projected/6f94a6d3-48dc-4cee-8af9-f8585b8fb973-kube-api-access-z9cpd\") pod \"node-resolver-5fs28\" (UID: \"6f94a6d3-48dc-4cee-8af9-f8585b8fb973\") " pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.180397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.180381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfbb\" (UniqueName: \"kubernetes.io/projected/035ca6b9-fb74-40ba-8dba-1c9043a36ed2-kube-api-access-sbfbb\") pod \"tuned-s5knm\" (UID: \"035ca6b9-fb74-40ba-8dba-1c9043a36ed2\") " pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.181645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.181625 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49nr\" (UniqueName: \"kubernetes.io/projected/1c47f7ce-8505-4759-b65e-759b36cf1034-kube-api-access-f49nr\") pod \"multus-s4mpv\" (UID: \"1c47f7ce-8505-4759-b65e-759b36cf1034\") " pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.181645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.181638 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn49q\" (UniqueName: \"kubernetes.io/projected/62cdebc6-f15a-4c45-85f0-f5c3b7ea7740-kube-api-access-cn49q\") pod \"aws-ebs-csi-driver-node-wkcr7\" (UID: \"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.277002 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.276910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5fs28" Jul 23 18:06:38.286237 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.286207 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f94a6d3_48dc_4cee_8af9_f8585b8fb973.slice/crio-0d1a9c70fdfbe34d67c4186f31f7bfa39e918fe915e24d794bf7dd1fbfa6bc85 WatchSource:0}: Error finding container 0d1a9c70fdfbe34d67c4186f31f7bfa39e918fe915e24d794bf7dd1fbfa6bc85: Status 404 returned error can't find the container with id 0d1a9c70fdfbe34d67c4186f31f7bfa39e918fe915e24d794bf7dd1fbfa6bc85 Jul 23 18:06:38.294505 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.294472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dd47g" Jul 23 18:06:38.301546 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.301512 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s4mpv" Jul 23 18:06:38.302061 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.302035 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba60a11b_ba4f_4987_ac74_481c32323c2a.slice/crio-2c67018fb5f7a038c5887db7290717580986ce4fd38ef2853e2d568e4725999f WatchSource:0}: Error finding container 2c67018fb5f7a038c5887db7290717580986ce4fd38ef2853e2d568e4725999f: Status 404 returned error can't find the container with id 2c67018fb5f7a038c5887db7290717580986ce4fd38ef2853e2d568e4725999f Jul 23 18:06:38.309133 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.309101 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c47f7ce_8505_4759_b65e_759b36cf1034.slice/crio-ff8bbab2a7e1bd2bc81c014017f08f00f56c0a975444a4780e0c8533cb04f506 WatchSource:0}: Error finding container ff8bbab2a7e1bd2bc81c014017f08f00f56c0a975444a4780e0c8533cb04f506: Status 404 returned error can't find the container with id ff8bbab2a7e1bd2bc81c014017f08f00f56c0a975444a4780e0c8533cb04f506 Jul 23 18:06:38.314875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.314854 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" Jul 23 18:06:38.320567 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.320544 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5knm" Jul 23 18:06:38.322496 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.322474 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cdebc6_f15a_4c45_85f0_f5c3b7ea7740.slice/crio-d60baf6dc0eae2ebef6388cb3dbd49f7de75990c7c045f3a15314a6587670d01 WatchSource:0}: Error finding container d60baf6dc0eae2ebef6388cb3dbd49f7de75990c7c045f3a15314a6587670d01: Status 404 returned error can't find the container with id d60baf6dc0eae2ebef6388cb3dbd49f7de75990c7c045f3a15314a6587670d01 Jul 23 18:06:38.328173 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.328141 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035ca6b9_fb74_40ba_8dba_1c9043a36ed2.slice/crio-82d122fa0a6aef442b9a64a657bb0fb5bff40671f06299cbce2c88825addb771 WatchSource:0}: Error finding container 82d122fa0a6aef442b9a64a657bb0fb5bff40671f06299cbce2c88825addb771: Status 404 returned error can't find the container with id 82d122fa0a6aef442b9a64a657bb0fb5bff40671f06299cbce2c88825addb771 Jul 23 18:06:38.346358 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.346320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbsxq" Jul 23 18:06:38.354073 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.354042 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4b3d0d_f43f_46a1_a999_440f0985c9ac.slice/crio-cfd97e353528c7bf38e909ecbd092ee1b93787903993e8c545899842a86561d3 WatchSource:0}: Error finding container cfd97e353528c7bf38e909ecbd092ee1b93787903993e8c545899842a86561d3: Status 404 returned error can't find the container with id cfd97e353528c7bf38e909ecbd092ee1b93787903993e8c545899842a86561d3 Jul 23 18:06:38.367541 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.367510 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tgdq9" Jul 23 18:06:38.374581 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.374557 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11788ed9_1469_46aa_a70e_cb063de2b14a.slice/crio-f4d2b8a84fc3294626aeac0437110f5073b7e86d2a0c2c47f5b29eebb2c44586 WatchSource:0}: Error finding container f4d2b8a84fc3294626aeac0437110f5073b7e86d2a0c2c47f5b29eebb2c44586: Status 404 returned error can't find the container with id f4d2b8a84fc3294626aeac0437110f5073b7e86d2a0c2c47f5b29eebb2c44586 Jul 23 18:06:38.381909 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.381889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:06:38.386940 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.386900 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:06:38.389474 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.389453 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cce2639_0478_49ca_a434_208bfc9dd501.slice/crio-d41e5f8ca7d26f793b78a7b7085215c527995b290052fc75a04a3e39f8b90298 WatchSource:0}: Error finding container d41e5f8ca7d26f793b78a7b7085215c527995b290052fc75a04a3e39f8b90298: Status 404 returned error can't find the container with id d41e5f8ca7d26f793b78a7b7085215c527995b290052fc75a04a3e39f8b90298 Jul 23 18:06:38.394888 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:06:38.394862 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8d624b_754c_45c4_98f6_848d0fc2f37b.slice/crio-fbc31c1c06d9548ef24b56f0d61b8acd1a35f09c06f190dd5f53b05e1f5f860f WatchSource:0}: Error finding container fbc31c1c06d9548ef24b56f0d61b8acd1a35f09c06f190dd5f53b05e1f5f860f: Status 404 returned error can't find the container with id fbc31c1c06d9548ef24b56f0d61b8acd1a35f09c06f190dd5f53b05e1f5f860f Jul 23 18:06:38.467933 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.467900 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:38.669500 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.669400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:38.669500 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.669487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:38.669706 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.669576 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:38.669706 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.669613 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:38.669706 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.669663 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:39.66963891 +0000 UTC m=+3.237236063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:38.669706 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.669681 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:39.669674056 +0000 UTC m=+3.237271204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:38.770701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.770021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:38.770701 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.770254 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:38.770701 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.770273 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:38.770701 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.770286 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:38.770701 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:38.770341 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:39.770324464 +0000 UTC m=+3.337921633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:38.961083 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:38.961004 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:39.006205 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.006136 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:01:37 +0000 UTC" deadline="2028-05-05 05:16:02.94409705 +0000 UTC" Jul 23 18:06:39.006205 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.006200 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15635h9m23.937901007s" Jul 23 18:06:39.139356 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.139324 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:39.139536 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.139450 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:39.149571 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.149486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fs28" event={"ID":"6f94a6d3-48dc-4cee-8af9-f8585b8fb973","Type":"ContainerStarted","Data":"0d1a9c70fdfbe34d67c4186f31f7bfa39e918fe915e24d794bf7dd1fbfa6bc85"} Jul 23 18:06:39.157501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.157394 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" event={"ID":"bd05e8e976bdb21bacf0ae5f21b68be6","Type":"ContainerStarted","Data":"6751997e34d79d895728443d413cbfdf5425918c31961246f20a0b42206d4b24"} Jul 23 18:06:39.164276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.164240 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"d41e5f8ca7d26f793b78a7b7085215c527995b290052fc75a04a3e39f8b90298"} Jul 23 18:06:39.170697 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.170572 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbsxq" event={"ID":"2e4b3d0d-f43f-46a1-a999-440f0985c9ac","Type":"ContainerStarted","Data":"cfd97e353528c7bf38e909ecbd092ee1b93787903993e8c545899842a86561d3"} Jul 23 18:06:39.176343 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.176262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5knm" event={"ID":"035ca6b9-fb74-40ba-8dba-1c9043a36ed2","Type":"ContainerStarted","Data":"82d122fa0a6aef442b9a64a657bb0fb5bff40671f06299cbce2c88825addb771"} Jul 23 18:06:39.183166 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.183104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" event={"ID":"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740","Type":"ContainerStarted","Data":"d60baf6dc0eae2ebef6388cb3dbd49f7de75990c7c045f3a15314a6587670d01"} Jul 23 18:06:39.187205 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.187171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerStarted","Data":"2c67018fb5f7a038c5887db7290717580986ce4fd38ef2853e2d568e4725999f"} Jul 23 18:06:39.199954 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.199897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" event={"ID":"1fc05c03729111c4ef810e639cee95d1","Type":"ContainerStarted","Data":"0858daa847b3d9075996765b719cbca048ca033bd0bd3241b930700b1127c4fc"} Jul 23 18:06:39.207983 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.207945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx84d" event={"ID":"5c8d624b-754c-45c4-98f6-848d0fc2f37b","Type":"ContainerStarted","Data":"fbc31c1c06d9548ef24b56f0d61b8acd1a35f09c06f190dd5f53b05e1f5f860f"} Jul 23 18:06:39.219208 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.218934 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tgdq9" event={"ID":"11788ed9-1469-46aa-a70e-cb063de2b14a","Type":"ContainerStarted","Data":"f4d2b8a84fc3294626aeac0437110f5073b7e86d2a0c2c47f5b29eebb2c44586"} Jul 23 18:06:39.232474 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.232435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s4mpv" event={"ID":"1c47f7ce-8505-4759-b65e-759b36cf1034","Type":"ContainerStarted","Data":"ff8bbab2a7e1bd2bc81c014017f08f00f56c0a975444a4780e0c8533cb04f506"} Jul 23 18:06:39.678568 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.678480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:39.678568 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.678566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:39.678834 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.678712 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:39.678834 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.678713 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:39.678834 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.678790 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:41.678771711 +0000 UTC m=+5.246368862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:39.678834 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.678809 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:41.678800543 +0000 UTC m=+5.246397696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:39.779986 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:39.779946 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:39.780172 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.780161 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:39.780235 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.780183 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:39.780235 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.780196 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:39.780352 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:39.780255 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:41.780235916 +0000 UTC m=+5.347833090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:40.006824 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.006714 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 18:01:37 +0000 UTC" deadline="2028-02-07 09:36:36.327615666 +0000 UTC" Jul 23 18:06:40.006824 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.006770 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h29m56.320851875s" Jul 23 18:06:40.137396 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.137365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:40.137588 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.137413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:40.137588 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:40.137506 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:40.137691 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:40.137644 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:40.214029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.213627 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:40.466701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:40.466619 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 18:06:41.137098 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:41.137022 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:41.137558 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.137161 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:41.698056 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:41.698013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:41.698251 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:41.698093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:41.698251 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.698240 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:41.698363 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.698303 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:45.698283121 +0000 UTC m=+9.265880284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:41.698707 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.698688 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:41.698810 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.698758 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:45.698728043 +0000 UTC m=+9.266325199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:41.799248 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:41.799211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:41.799441 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.799374 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:41.799441 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.799393 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:41.799441 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.799406 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:41.799603 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:41.799480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:45.799460915 +0000 UTC m=+9.367058066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:42.137325 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:42.137170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:42.137823 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:42.137360 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:42.137823 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:42.137417 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:42.137823 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:42.137528 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:42.251311 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:42.250820 2570 generic.go:356] "Generic (PLEG): container finished" podID="1fc05c03729111c4ef810e639cee95d1" containerID="3cf2deb65b2f2226c86447beec5150c9d10b59a27465fc970fb705079dea9291" exitCode=0 Jul 23 18:06:42.251311 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:42.250871 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" event={"ID":"1fc05c03729111c4ef810e639cee95d1","Type":"ContainerDied","Data":"3cf2deb65b2f2226c86447beec5150c9d10b59a27465fc970fb705079dea9291"} Jul 23 18:06:43.139438 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:43.139406 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:43.139936 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:43.139514 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:44.137415 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:44.137342 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:44.137585 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:44.137482 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:44.137660 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:44.137630 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:44.137788 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:44.137762 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:45.137449 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:45.137370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:45.137923 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.137517 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:45.734631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:45.734727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.734877 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.734949 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:53.734921887 +0000 UTC m=+17.302519038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.735000 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:45.735167 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.735074 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:53.735054195 +0000 UTC m=+17.302651358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:45.835475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:45.835425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:45.835678 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.835661 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:45.835781 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.835684 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:45.835781 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.835698 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:45.835899 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:45.835789 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:06:53.835768244 +0000 UTC m=+17.403365416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:46.137495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:46.137414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:46.137954 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:46.137564 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:46.137954 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:46.137574 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:46.137954 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:46.137668 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:47.138388 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:47.137898 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:47.138388 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:47.138018 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:48.136730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:48.136694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:48.136973 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:48.136859 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:48.137211 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:48.136694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:48.137211 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:48.137170 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:49.136990 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:49.136953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:49.137443 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:49.137087 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:50.137291 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:50.137243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:50.137679 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:50.137244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:50.137679 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:50.137359 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:50.137679 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:50.137470 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:51.137256 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:51.137219 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:51.137442 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:51.137376 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:52.136739 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:52.136703 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:52.136920 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:52.136900 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:52.136989 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:52.136969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:52.137101 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:52.137083 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:53.137135 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:53.137097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:53.137608 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.137246 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:53.795397 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:53.795360 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:53.795647 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:53.795444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:53.795647 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.795544 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:53.795647 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.795620 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:09.795597782 +0000 UTC m=+33.363194933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:06:53.795647 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.795548 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:53.795874 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.795687 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:09.79567437 +0000 UTC m=+33.363271531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:06:53.896029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:53.895988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:53.896237 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.896133 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:06:53.896237 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.896153 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:06:53.896237 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.896167 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:53.896237 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:53.896223 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:09.896206305 +0000 UTC m=+33.463803468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:06:54.137337 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:54.137244 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:54.137337 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:54.137281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:54.137884 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:54.137392 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:54.137884 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:54.137559 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:55.137374 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:55.137338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:55.137828 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:55.137471 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:56.136706 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:56.136665 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:56.136706 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:56.136702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:56.136961 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:56.136824 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:56.137019 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:56.136979 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:57.137683 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:57.137641 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:57.138150 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:57.137760 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:58.137487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:58.137445 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:06:58.137487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:58.137475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:06:58.137763 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:58.137586 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:06:58.137763 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:58.137665 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:06:59.137469 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.137311 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:06:59.137597 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:06:59.137546 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:06:59.285870 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.285836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" event={"ID":"1fc05c03729111c4ef810e639cee95d1","Type":"ContainerStarted","Data":"84f56df82b23cff7164e5a7d705a4754a06d63f74c99c2c2432a1d10c063e583"} Jul 23 18:06:59.287216 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.287186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bx84d" event={"ID":"5c8d624b-754c-45c4-98f6-848d0fc2f37b","Type":"ContainerStarted","Data":"01169bdd2d6b5f5698798c14fe76618db3ab8f5b336e0102121d1de5fff945f7"} Jul 23 18:06:59.288512 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.288483 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s4mpv" event={"ID":"1c47f7ce-8505-4759-b65e-759b36cf1034","Type":"ContainerStarted","Data":"22e509c8cfb7953652aa9a2d1fc51b056448aa97894dd94c2a112f38098034be"} Jul 23 18:06:59.289735 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.289716 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5fs28" event={"ID":"6f94a6d3-48dc-4cee-8af9-f8585b8fb973","Type":"ContainerStarted","Data":"c3690d718e0267474bac3407ec5f6df47cc96c70112e979f3af264afca5a3ef4"} Jul 23 18:06:59.291046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.291026 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" event={"ID":"bd05e8e976bdb21bacf0ae5f21b68be6","Type":"ContainerStarted","Data":"b924386dbd33bb21d0469e592d8e3ca0f065ca86ef7deb7bc7cfd876f17226ed"} Jul 23 18:06:59.292500 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.292483 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:06:59.292816 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.292797 2570 generic.go:356] "Generic (PLEG): container finished" podID="9cce2639-0478-49ca-a434-208bfc9dd501" containerID="d01a9ee3a61fdd8a2cabb136eccc5e6817612d435a1bd60e161b22307bc1e0fd" exitCode=1 Jul 23 18:06:59.292891 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.292867 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerDied","Data":"d01a9ee3a61fdd8a2cabb136eccc5e6817612d435a1bd60e161b22307bc1e0fd"} Jul 23 18:06:59.292936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.292922 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"d901a12edb02abbf4419370a38b48137ffa0e0185003d3e598e4517d47f0c33e"} Jul 23 18:06:59.294174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.294150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbsxq" event={"ID":"2e4b3d0d-f43f-46a1-a999-440f0985c9ac","Type":"ContainerStarted","Data":"26b0356f1b312ce90f85742be3890ee84d7e283500357f9eef80cac6e9182deb"} Jul 23 18:06:59.295525 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.295497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5knm" event={"ID":"035ca6b9-fb74-40ba-8dba-1c9043a36ed2","Type":"ContainerStarted","Data":"32e4a75587095a52bd334da62738d0d746c5903bb2817a9e99773641dc5e302b"} Jul 23 18:06:59.297135 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.297115 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" event={"ID":"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740","Type":"ContainerStarted","Data":"b7ad4c4b1f058be2152e6effc659b791b7dcdb5eae89a68b31db3c7b2b969f75"} Jul 23 18:06:59.298415 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.298395 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="24afa2d148ef92083ee0cffe42d8b9923e2f1109d94e918a09e8a603b8380d25" exitCode=0 Jul 23 18:06:59.298507 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.298422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"24afa2d148ef92083ee0cffe42d8b9923e2f1109d94e918a09e8a603b8380d25"} Jul 23 18:06:59.329429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.329344 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bx84d" podStartSLOduration=10.824407466 podStartE2EDuration="22.329322013s" totalImagesPullingTime="11.504914547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.328853508 +0000 UTC m=+22.896450680" watchObservedRunningTime="2026-07-23 18:06:59.329322013 +0000 UTC m=+22.896919184" Jul 23 18:06:59.329853 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.329816 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-197.ec2.internal" podStartSLOduration=22.329799505 podStartE2EDuration="22.329799505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.308290239 +0000 UTC m=+22.875887410" watchObservedRunningTime="2026-07-23 18:06:59.329799505 +0000 UTC m=+22.897396680" Jul 23 18:06:59.383209 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.382891 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-197.ec2.internal" podStartSLOduration=22.382846591 podStartE2EDuration="22.382846591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.35165558 +0000 UTC m=+22.919252750" watchObservedRunningTime="2026-07-23 18:06:59.382846591 +0000 UTC m=+22.950443762" Jul 23 18:06:59.466120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.466062 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5fs28" podStartSLOduration=2.087106933 podStartE2EDuration="22.466039818s" totalImagesPullingTime="20.378932885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.38405332 +0000 UTC m=+22.951650492" watchObservedRunningTime="2026-07-23 18:06:59.466039818 +0000 UTC m=+23.033636990" Jul 23 18:06:59.526460 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.526416 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s5knm" podStartSLOduration=2.189181054 podStartE2EDuration="22.52639969s" totalImagesPullingTime="20.337218636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.492527894 +0000 UTC m=+23.060125065" watchObservedRunningTime="2026-07-23 18:06:59.52639969 +0000 UTC m=+23.093996860" Jul 23 18:06:59.562543 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.562490 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s4mpv" podStartSLOduration=2.15838106 podStartE2EDuration="22.562476554s" totalImagesPullingTime="20.404095494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.561997614 +0000 UTC m=+23.129594786" watchObservedRunningTime="2026-07-23 18:06:59.562476554 +0000 UTC m=+23.130073725" Jul 23 18:06:59.562812 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:06:59.562791 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wbsxq" podStartSLOduration=2.32684346 podStartE2EDuration="22.562786262s" totalImagesPullingTime="20.235942802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:06:59.526907337 +0000 UTC m=+23.094504507" watchObservedRunningTime="2026-07-23 18:06:59.562786262 +0000 UTC m=+23.130383432" Jul 23 18:07:00.136673 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.136645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:00.136673 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.136664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:00.136889 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:00.136773 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:00.136948 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:00.136876 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:00.150621 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.150590 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 18:07:00.302354 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.302256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tgdq9" event={"ID":"11788ed9-1469-46aa-a70e-cb063de2b14a","Type":"ContainerStarted","Data":"6ccc520fbb5fdff9963806e73bd03be1178386ba73de10eb3aca485cfc56798b"} Jul 23 18:07:00.305626 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.305605 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:07:00.306047 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.306021 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"1ee3b63c0eb5e10d9aa58a0ac050a867482c6c32deeb4c9e09bc7599d8b0ce5d"} Jul 23 18:07:00.306170 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.306096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"ab510ccb0d803c43697188607d710320d18f474396964d7e5f2a8abc216cf898"} Jul 23 18:07:00.306170 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.306150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"fbc0dd72e073f479617ce38cd3d816b82b014227cfde3c3219b27e6b697312c7"} Jul 23 18:07:00.306280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.306197 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"eebffd1aac62ff6f878a06746850b302b97edf361aa6e57ecef400e9af1b01ef"} Jul 23 18:07:00.308121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:00.308093 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" event={"ID":"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740","Type":"ContainerStarted","Data":"2d2977122820419a479b79cf0d13470a59c58584297637c66c26272b33d6e04d"} Jul 23 18:07:01.026778 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.026650 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T18:07:00.150614074Z","UUID":"eb94295e-1aa5-4a71-9101-da646baeb555","Handler":null,"Name":"","Endpoint":""} Jul 23 18:07:01.028343 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.028314 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 18:07:01.028488 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.028364 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 18:07:01.100224 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.100184 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:07:01.101382 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.101359 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:07:01.132635 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.132578 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tgdq9" podStartSLOduration=3.9171868610000002 podStartE2EDuration="24.132560614s" totalImagesPullingTime="20.215373753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:00.32983524 +0000 UTC m=+23.897432410" watchObservedRunningTime="2026-07-23 18:07:01.132560614 +0000 UTC m=+24.700157787" Jul 23 18:07:01.137180 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.137151 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:01.137362 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:01.137323 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:01.311392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.311243 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:07:01.311879 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:01.311629 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bx84d" Jul 23 18:07:02.136788 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.136733 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:02.137028 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.136771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:02.137028 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:02.136910 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:02.137028 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:02.136980 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:02.317010 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.316934 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:07:02.317521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.317411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"f5d7f1e4062301b849595add25bc709573a2f6e3ec8a49be0c948ce9df1c57a5"} Jul 23 18:07:02.319882 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.319850 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" event={"ID":"62cdebc6-f15a-4c45-85f0-f5c3b7ea7740","Type":"ContainerStarted","Data":"c46dba2c7b3b28d9c59e97b977af6a3ad83d7770076de6bc44dadc8bd9ac5025"} Jul 23 18:07:02.344831 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:02.344776 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wkcr7" podStartSLOduration=2.180402566 podStartE2EDuration="25.344760069s" totalImagesPullingTime="23.164357503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:02.344552621 +0000 UTC m=+25.912149792" watchObservedRunningTime="2026-07-23 18:07:02.344760069 +0000 UTC m=+25.912357230" Jul 23 18:07:03.137538 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:03.137301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:03.137773 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:03.137652 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:04.137286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.137246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:04.138001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.137246 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:04.138001 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:04.137358 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:04.138001 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:04.137482 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:04.328636 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.328401 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:07:04.328954 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.328925 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"83752064daa9ea0ad25001bae937f01764334344c8b739e754839b51c956d261"} Jul 23 18:07:04.329193 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.329163 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:04.329426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.329408 2570 scope.go:117] "RemoveContainer" containerID="d01a9ee3a61fdd8a2cabb136eccc5e6817612d435a1bd60e161b22307bc1e0fd" Jul 23 18:07:04.331077 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.331048 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="7a0ceb8ececd919dec6582bd3258249a582bb69c917533dd5cf2e8665df86193" exitCode=0 Jul 23 18:07:04.331181 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.331102 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"7a0ceb8ececd919dec6582bd3258249a582bb69c917533dd5cf2e8665df86193"} Jul 23 18:07:04.347176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:04.347154 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:05.137134 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.137058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:05.137311 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:05.137170 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:05.343463 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.343435 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:07:05.343886 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.343847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" event={"ID":"9cce2639-0478-49ca-a434-208bfc9dd501","Type":"ContainerStarted","Data":"4222bf5a3e0189ef308cec919bba959a84dd6c644a7b41b8f2366b832fb1fe34"} Jul 23 18:07:05.344207 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.344180 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:05.344323 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.344253 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:05.363409 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.363373 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:05.477253 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:05.477150 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" podStartSLOduration=7.9216506209999995 podStartE2EDuration="28.477132987s" totalImagesPullingTime="20.555482366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:05.418135084 +0000 UTC m=+28.985732272" watchObservedRunningTime="2026-07-23 18:07:05.477132987 +0000 UTC m=+29.044730158" Jul 23 18:07:06.136923 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.136893 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:06.136923 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.136893 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:06.137230 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:06.136998 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:06.137230 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:06.137137 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:06.349513 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.349472 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="c957ca1ffbc588d44ef7dcf13333f5222078154c118099c94e3aa07a5d6afc5c" exitCode=0 Jul 23 18:07:06.350172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.350147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"c957ca1ffbc588d44ef7dcf13333f5222078154c118099c94e3aa07a5d6afc5c"} Jul 23 18:07:06.634631 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.634595 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bmskp"] Jul 23 18:07:06.634831 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.634707 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:06.634831 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:06.634805 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:06.649583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.649496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x2wcs"] Jul 23 18:07:06.649767 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.649629 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:06.649846 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:06.649784 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:06.650144 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.650121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w79t6"] Jul 23 18:07:06.650237 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:06.650225 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:06.650336 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:06.650317 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:08.137399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:08.137207 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:08.137816 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:08.137222 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:08.137816 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:08.137471 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:08.137816 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:08.137289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:08.137816 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:08.137555 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:08.137816 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:08.137645 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:08.356429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:08.356391 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="df082063740c128746139200383684d97ebfca5c0887a6085f3123f544ee99d0" exitCode=0 Jul 23 18:07:08.356783 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:08.356436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"df082063740c128746139200383684d97ebfca5c0887a6085f3123f544ee99d0"} Jul 23 18:07:09.814071 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:09.814035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:09.814693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:09.814123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:09.814693 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.814193 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:07:09.814693 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.814257 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 18:07:09.814693 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.814266 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs podName:114cc0ef-9e50-4957-95ba-44398bf8de22 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:41.814250067 +0000 UTC m=+65.381847217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs") pod "network-metrics-daemon-x2wcs" (UID: "114cc0ef-9e50-4957-95ba-44398bf8de22") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 18:07:09.814693 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.814333 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret podName:b766b43e-8eb4-43bd-9671-ed7c0c3a0e98 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:41.814315308 +0000 UTC m=+65.381912471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret") pod "global-pull-secret-syncer-bmskp" (UID: "b766b43e-8eb4-43bd-9671-ed7c0c3a0e98") : object "kube-system"/"original-pull-secret" not registered Jul 23 18:07:09.915212 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:09.915177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:09.915380 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.915339 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 18:07:09.915380 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.915359 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 18:07:09.915380 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.915368 2570 projected.go:196] Error preparing data for projected volume kube-api-access-scbvz for pod openshift-network-diagnostics/network-check-target-w79t6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:07:09.915539 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:09.915421 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz podName:d21a8d63-8fb6-413c-bb9d-2c59b2f15c13 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:41.915404962 +0000 UTC m=+65.483002119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-scbvz" (UniqueName: "kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz") pod "network-check-target-w79t6" (UID: "d21a8d63-8fb6-413c-bb9d-2c59b2f15c13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 18:07:10.137304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:10.137213 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:10.137304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:10.137247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:10.137304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:10.137247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:10.137585 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:10.137375 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:10.137585 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:10.137471 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:10.137585 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:10.137559 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:12.136591 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.136553 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:12.137288 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:12.136674 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w79t6" podUID="d21a8d63-8fb6-413c-bb9d-2c59b2f15c13" Jul 23 18:07:12.137288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.136992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:12.137288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.137058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:12.137288 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:12.137092 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bmskp" podUID="b766b43e-8eb4-43bd-9671-ed7c0c3a0e98" Jul 23 18:07:12.137288 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:12.137178 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x2wcs" podUID="114cc0ef-9e50-4957-95ba-44398bf8de22" Jul 23 18:07:12.745673 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.745635 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-197.ec2.internal" event="NodeReady" Jul 23 18:07:12.745882 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.745836 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 18:07:12.815335 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.815300 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l8sxs"] Jul 23 18:07:12.829755 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.829721 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-smfl5"] Jul 23 18:07:12.829915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.829842 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:12.833787 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.833570 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtn54\"" Jul 23 18:07:12.833787 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.833691 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 18:07:12.834249 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.834032 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 18:07:12.834249 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.834089 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 18:07:12.852578 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.852540 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l8sxs"] Jul 23 18:07:12.852578 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.852580 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-smfl5"] Jul 23 18:07:12.852808 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.852657 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:12.855511 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.855493 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 18:07:12.855998 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.855982 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 18:07:12.856059 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.856045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kfp27\"" Jul 23 18:07:12.937174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xdk\" (UniqueName: \"kubernetes.io/projected/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-kube-api-access-r9xdk\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:12.937174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0701e9-aee0-41c9-9900-fc2d46bfd966-config-volume\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:12.937403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937200 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb0701e9-aee0-41c9-9900-fc2d46bfd966-tmp-dir\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:12.937403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937225 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:12.937403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937332 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr695\" (UniqueName: \"kubernetes.io/projected/cb0701e9-aee0-41c9-9900-fc2d46bfd966-kube-api-access-dr695\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:12.937403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:12.937365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-cert\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.038372 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xdk\" (UniqueName: \"kubernetes.io/projected/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-kube-api-access-r9xdk\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.038372 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0701e9-aee0-41c9-9900-fc2d46bfd966-config-volume\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.038372 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb0701e9-aee0-41c9-9900-fc2d46bfd966-tmp-dir\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.038598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.038598 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:13.038489 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:07:13.038598 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:13.038551 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls podName:cb0701e9-aee0-41c9-9900-fc2d46bfd966 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:13.538533418 +0000 UTC m=+37.106130591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls") pod "dns-default-smfl5" (UID: "cb0701e9-aee0-41c9-9900-fc2d46bfd966") : secret "dns-default-metrics-tls" not found Jul 23 18:07:13.038598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr695\" (UniqueName: \"kubernetes.io/projected/cb0701e9-aee0-41c9-9900-fc2d46bfd966-kube-api-access-dr695\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.038598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-cert\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.038854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038791 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb0701e9-aee0-41c9-9900-fc2d46bfd966-tmp-dir\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.038957 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.038936 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0701e9-aee0-41c9-9900-fc2d46bfd966-config-volume\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.042836 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.042817 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-cert\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.048891 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.048870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xdk\" (UniqueName: \"kubernetes.io/projected/2cee312f-0de4-436b-bd3c-21e0ed6f93d9-kube-api-access-r9xdk\") pod \"ingress-canary-l8sxs\" (UID: \"2cee312f-0de4-436b-bd3c-21e0ed6f93d9\") " pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.049510 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.049492 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr695\" (UniqueName: \"kubernetes.io/projected/cb0701e9-aee0-41c9-9900-fc2d46bfd966-kube-api-access-dr695\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.140034 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.140007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l8sxs" Jul 23 18:07:13.352866 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.352614 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l8sxs"] Jul 23 18:07:13.357339 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:13.357311 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cee312f_0de4_436b_bd3c_21e0ed6f93d9.slice/crio-02cd01e45f8b6db6e164af22202e5882176a668a5e5805cf2e25f35e43b31b73 WatchSource:0}: Error finding container 02cd01e45f8b6db6e164af22202e5882176a668a5e5805cf2e25f35e43b31b73: Status 404 returned error can't find the container with id 02cd01e45f8b6db6e164af22202e5882176a668a5e5805cf2e25f35e43b31b73 Jul 23 18:07:13.370912 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.370874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l8sxs" event={"ID":"2cee312f-0de4-436b-bd3c-21e0ed6f93d9","Type":"ContainerStarted","Data":"02cd01e45f8b6db6e164af22202e5882176a668a5e5805cf2e25f35e43b31b73"} Jul 23 18:07:13.374224 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.374196 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerStarted","Data":"dc663a1f4fbee1878b0e185282191b27084e50aaef58c04056d07bc6386fda97"} Jul 23 18:07:13.542568 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:13.542460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:13.542721 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:13.542594 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:07:13.542721 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:13.542656 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls podName:cb0701e9-aee0-41c9-9900-fc2d46bfd966 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:14.542641914 +0000 UTC m=+38.110239064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls") pod "dns-default-smfl5" (UID: "cb0701e9-aee0-41c9-9900-fc2d46bfd966") : secret "dns-default-metrics-tls" not found Jul 23 18:07:14.137467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.137425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:14.137659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.137477 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:14.137659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.137477 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:14.147103 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.147048 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 18:07:14.150522 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.149124 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 18:07:14.150522 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.149405 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 18:07:14.150522 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.149939 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 18:07:14.150522 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.150369 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bbklj\"" Jul 23 18:07:14.150845 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.150655 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t929l\"" Jul 23 18:07:14.381034 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.380992 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="dc663a1f4fbee1878b0e185282191b27084e50aaef58c04056d07bc6386fda97" exitCode=0 Jul 23 18:07:14.381186 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.381085 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"dc663a1f4fbee1878b0e185282191b27084e50aaef58c04056d07bc6386fda97"} Jul 23 18:07:14.553599 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.553559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:14.553770 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:14.553735 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:07:14.553845 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:14.553822 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls podName:cb0701e9-aee0-41c9-9900-fc2d46bfd966 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:16.553800191 +0000 UTC m=+40.121397358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls") pod "dns-default-smfl5" (UID: "cb0701e9-aee0-41c9-9900-fc2d46bfd966") : secret "dns-default-metrics-tls" not found Jul 23 18:07:14.671392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.671323 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs"] Jul 23 18:07:14.673199 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.673173 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.678093 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.678071 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 18:07:14.681923 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.681898 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 18:07:14.681923 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.681917 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 18:07:14.682092 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.682026 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 18:07:14.693884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.693794 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs"] Jul 23 18:07:14.711844 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.711811 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc"] Jul 23 18:07:14.714065 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.714041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.723406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.722919 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 18:07:14.723406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.723184 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 18:07:14.723975 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.723951 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 18:07:14.730761 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.730725 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 18:07:14.738718 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.738694 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc"] Jul 23 18:07:14.755416 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.755382 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b2426a2-0921-4b86-8f68-7602a90ff7b5-tmp\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.755577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.755427 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b2426a2-0921-4b86-8f68-7602a90ff7b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.755577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.755472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgflh\" (UniqueName: \"kubernetes.io/projected/9b2426a2-0921-4b86-8f68-7602a90ff7b5-kube-api-access-jgflh\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.856264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b2426a2-0921-4b86-8f68-7602a90ff7b5-tmp\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b2426a2-0921-4b86-8f68-7602a90ff7b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856349 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856433 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx6g\" (UniqueName: \"kubernetes.io/projected/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-kube-api-access-rwx6g\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856712 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgflh\" (UniqueName: \"kubernetes.io/projected/9b2426a2-0921-4b86-8f68-7602a90ff7b5-kube-api-access-jgflh\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.856712 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856490 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.856827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.856798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b2426a2-0921-4b86-8f68-7602a90ff7b5-tmp\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.860363 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.860339 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9b2426a2-0921-4b86-8f68-7602a90ff7b5-klusterlet-config\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.868054 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.868023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgflh\" (UniqueName: \"kubernetes.io/projected/9b2426a2-0921-4b86-8f68-7602a90ff7b5-kube-api-access-jgflh\") pod \"klusterlet-addon-workmgr-594775674d-dgmzs\" (UID: \"9b2426a2-0921-4b86-8f68-7602a90ff7b5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:14.957938 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.957856 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.957938 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.957902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.958143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.957942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.958143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.957988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx6g\" (UniqueName: \"kubernetes.io/projected/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-kube-api-access-rwx6g\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.958143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.958023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.958143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.958105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.958925 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.958857 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.961310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.961287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.961425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.961336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-ca\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.961425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.961370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.961425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.961370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.982726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.982683 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx6g\" (UniqueName: \"kubernetes.io/projected/f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9-kube-api-access-rwx6g\") pod \"cluster-proxy-proxy-agent-69584f9df7-phjxc\" (UID: \"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:14.983297 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:14.983270 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:15.031324 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:15.031288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:07:15.387008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:15.386795 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba60a11b-ba4f-4987-ac74-481c32323c2a" containerID="51a7b5b99a1ea911a8af4a39956b8931b89e7b7b05d6539e8b24a91763684387" exitCode=0 Jul 23 18:07:15.387477 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:15.386884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerDied","Data":"51a7b5b99a1ea911a8af4a39956b8931b89e7b7b05d6539e8b24a91763684387"} Jul 23 18:07:15.803613 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:15.803571 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs"] Jul 23 18:07:15.811933 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:15.811882 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2426a2_0921_4b86_8f68_7602a90ff7b5.slice/crio-d9c3e528ef36ab8aa5113cd69e0d590306ffd96bd491e096d9344c4d05ad28d9 WatchSource:0}: Error finding container d9c3e528ef36ab8aa5113cd69e0d590306ffd96bd491e096d9344c4d05ad28d9: Status 404 returned error can't find the container with id d9c3e528ef36ab8aa5113cd69e0d590306ffd96bd491e096d9344c4d05ad28d9 Jul 23 18:07:15.823493 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:15.823462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc"] Jul 23 18:07:15.826639 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:15.826609 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf335e3c1_3fcc_4af2_82dc_8f6aeccf92a9.slice/crio-18641d88e14c65bacf05f2df53f40df99b5339e98a71ead408439e391d9d94d9 WatchSource:0}: Error finding container 18641d88e14c65bacf05f2df53f40df99b5339e98a71ead408439e391d9d94d9: Status 404 returned error can't find the container with id 18641d88e14c65bacf05f2df53f40df99b5339e98a71ead408439e391d9d94d9 Jul 23 18:07:16.391627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.391589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l8sxs" event={"ID":"2cee312f-0de4-436b-bd3c-21e0ed6f93d9","Type":"ContainerStarted","Data":"a70e830ae3e0477b5225389fd2a3743381fbffc638984ff21d067456ed00f7d9"} Jul 23 18:07:16.392911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.392890 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerStarted","Data":"18641d88e14c65bacf05f2df53f40df99b5339e98a71ead408439e391d9d94d9"} Jul 23 18:07:16.396007 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.395984 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dd47g" event={"ID":"ba60a11b-ba4f-4987-ac74-481c32323c2a","Type":"ContainerStarted","Data":"f6bf15229beda291b8e44096d4ca7237c409647c6d86acd5b473dedd5d0b7922"} Jul 23 18:07:16.397007 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.396984 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" event={"ID":"9b2426a2-0921-4b86-8f68-7602a90ff7b5","Type":"ContainerStarted","Data":"d9c3e528ef36ab8aa5113cd69e0d590306ffd96bd491e096d9344c4d05ad28d9"} Jul 23 18:07:16.425118 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.425025 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l8sxs" podStartSLOduration=2.125230483 podStartE2EDuration="4.425010111s" totalImagesPullingTime="2.299779628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:16.424547653 +0000 UTC m=+39.992144843" watchObservedRunningTime="2026-07-23 18:07:16.425010111 +0000 UTC m=+39.992607282" Jul 23 18:07:16.477139 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.477082 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dd47g" podStartSLOduration=7.234030756 podStartE2EDuration="39.477060631s" totalImagesPullingTime="32.243029875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:16.476532304 +0000 UTC m=+40.044129488" watchObservedRunningTime="2026-07-23 18:07:16.477060631 +0000 UTC m=+40.044657815" Jul 23 18:07:16.572127 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:16.572085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:16.572327 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:16.572238 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:07:16.572327 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:16.572313 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls podName:cb0701e9-aee0-41c9-9900-fc2d46bfd966 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:20.572292556 +0000 UTC m=+44.139889742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls") pod "dns-default-smfl5" (UID: "cb0701e9-aee0-41c9-9900-fc2d46bfd966") : secret "dns-default-metrics-tls" not found Jul 23 18:07:17.450852 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.450794 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5v9vc"] Jul 23 18:07:17.456426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.455728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.460243 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.459997 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 18:07:17.460243 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.460182 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 18:07:17.461804 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.461711 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 18:07:17.461804 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.461761 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j5mk2\"" Jul 23 18:07:17.462289 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.462231 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 18:07:17.481888 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.481850 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v9vc"] Jul 23 18:07:17.581919 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.581879 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.582066 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.581927 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-api-access-sdphh\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.582066 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.581960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.582066 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.582061 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-data-volume\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.582203 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.582094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-crio-socket\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.683215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-data-volume\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.683266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-crio-socket\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.683356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.683386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-api-access-sdphh\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.683417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:17.683616 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:17.683688 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:18.183668329 +0000 UTC m=+41.751265482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : configmap "kube-rbac-proxy" not found Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:17.684036 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:17.684093 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:18.18407736 +0000 UTC m=+41.751674528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : secret "insights-runtime-extractor-tls" not found Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.684256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-crio-socket\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.684328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.684282 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-data-volume\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:17.700731 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:17.700698 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdphh\" (UniqueName: \"kubernetes.io/projected/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-api-access-sdphh\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:18.188382 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:18.188342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:18.188571 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:18.188398 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:18.188571 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:18.188483 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:07:18.188571 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:18.188554 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:19.188535724 +0000 UTC m=+42.756132878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : configmap "kube-rbac-proxy" not found Jul 23 18:07:18.188571 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:18.188483 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:07:18.188891 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:18.188633 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:19.188617497 +0000 UTC m=+42.756214655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : secret "insights-runtime-extractor-tls" not found Jul 23 18:07:19.196008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:19.195964 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:19.196008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:19.196011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:19.196516 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:19.196134 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 18:07:19.196516 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:19.196146 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:07:19.196516 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:19.196210 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:21.196190328 +0000 UTC m=+44.763787489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : configmap "kube-rbac-proxy" not found Jul 23 18:07:19.196516 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:19.196248 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:21.196219168 +0000 UTC m=+44.763816321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : secret "insights-runtime-extractor-tls" not found Jul 23 18:07:19.431015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:19.430986 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fs28_6f94a6d3-48dc-4cee-8af9-f8585b8fb973/dns-node-resolver/0.log" Jul 23 18:07:20.431222 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:20.431190 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbsxq_2e4b3d0d-f43f-46a1-a999-440f0985c9ac/node-ca/0.log" Jul 23 18:07:20.608804 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:20.608769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:20.609000 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:20.608933 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 18:07:20.609062 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:20.609011 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls podName:cb0701e9-aee0-41c9-9900-fc2d46bfd966 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:28.608991499 +0000 UTC m=+52.176588651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls") pod "dns-default-smfl5" (UID: "cb0701e9-aee0-41c9-9900-fc2d46bfd966") : secret "dns-default-metrics-tls" not found Jul 23 18:07:21.214456 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:21.214419 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:21.214670 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:21.214474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:21.214670 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:21.214535 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:07:21.214670 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:21.214614 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:25.214594764 +0000 UTC m=+48.782191936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : secret "insights-runtime-extractor-tls" not found Jul 23 18:07:21.215156 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:21.215134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:21.599844 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:21.599808 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l8sxs_2cee312f-0de4-436b-bd3c-21e0ed6f93d9/serve-healthcheck-canary/0.log" Jul 23 18:07:22.414622 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.414579 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" event={"ID":"9b2426a2-0921-4b86-8f68-7602a90ff7b5","Type":"ContainerStarted","Data":"c59cee2e31a78296aa116cb33675c3674a2870d950387e8bb3375d9f00f5bc5e"} Jul 23 18:07:22.414913 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.414670 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:22.416617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.416585 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" Jul 23 18:07:22.416933 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.416911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerStarted","Data":"ba7b776d34a9cdd758b2430e333570a285e407f9d3a2be6a6ff27d6247e3ac08"} Jul 23 18:07:22.416987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.416944 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerStarted","Data":"aadd4cec076e6bf784336e99e21876156b45543c873830f655c7f4bf6a60d84f"} Jul 23 18:07:22.416987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.416960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerStarted","Data":"06e2febaab9a35808fdc53b453309b9154d18990d6ec4d65c08be67a90f16b0f"} Jul 23 18:07:22.482296 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.482242 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-594775674d-dgmzs" podStartSLOduration=2.517644764 podStartE2EDuration="8.48222422s" totalImagesPullingTime="5.964579456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:22.452291455 +0000 UTC m=+46.019888626" watchObservedRunningTime="2026-07-23 18:07:22.48222422 +0000 UTC m=+46.049821369" Jul 23 18:07:22.524362 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:22.524311 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" podStartSLOduration=2.562653364 podStartE2EDuration="8.524293851s" totalImagesPullingTime="5.961640487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:22.52409273 +0000 UTC m=+46.091689900" watchObservedRunningTime="2026-07-23 18:07:22.524293851 +0000 UTC m=+46.091891021" Jul 23 18:07:25.248967 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:25.248927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:25.249347 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:25.249090 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 18:07:25.249347 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:25.249151 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls podName:f1ac5912-aaf9-4eb0-8212-fb607a65fdc9 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:33.249134653 +0000 UTC m=+56.816731818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v9vc" (UID: "f1ac5912-aaf9-4eb0-8212-fb607a65fdc9") : secret "insights-runtime-extractor-tls" not found Jul 23 18:07:28.672722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:28.672687 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:28.675002 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:28.674977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb0701e9-aee0-41c9-9900-fc2d46bfd966-metrics-tls\") pod \"dns-default-smfl5\" (UID: \"cb0701e9-aee0-41c9-9900-fc2d46bfd966\") " pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:28.761085 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:28.761045 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:28.909575 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:28.909532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-smfl5"] Jul 23 18:07:28.913424 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:28.913390 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0701e9_aee0_41c9_9900_fc2d46bfd966.slice/crio-04bb203954550911f8f91ed78d10c67070ed883de9f47fd926728b257287d5b1 WatchSource:0}: Error finding container 04bb203954550911f8f91ed78d10c67070ed883de9f47fd926728b257287d5b1: Status 404 returned error can't find the container with id 04bb203954550911f8f91ed78d10c67070ed883de9f47fd926728b257287d5b1 Jul 23 18:07:29.435420 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:29.435391 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-smfl5" event={"ID":"cb0701e9-aee0-41c9-9900-fc2d46bfd966","Type":"ContainerStarted","Data":"04bb203954550911f8f91ed78d10c67070ed883de9f47fd926728b257287d5b1"} Jul 23 18:07:31.442418 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:31.442387 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-smfl5" event={"ID":"cb0701e9-aee0-41c9-9900-fc2d46bfd966","Type":"ContainerStarted","Data":"13e467f38431b3923ef0216e77ae8663097ffaef91c9ff558416d452f6100c31"} Jul 23 18:07:32.447269 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:32.447235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-smfl5" event={"ID":"cb0701e9-aee0-41c9-9900-fc2d46bfd966","Type":"ContainerStarted","Data":"3d5c5539f4a6b4e31a036d074c78941229be95b6b9ea8162ff2e2ccb7aba99d0"} Jul 23 18:07:32.447674 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:32.447370 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:32.478617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:32.478561 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-smfl5" podStartSLOduration=18.08514857 podStartE2EDuration="20.478544034s" totalImagesPullingTime="2.393395464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:32.478005985 +0000 UTC m=+56.045603157" watchObservedRunningTime="2026-07-23 18:07:32.478544034 +0000 UTC m=+56.046141204" Jul 23 18:07:33.305587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:33.305535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:33.307810 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:33.307790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1ac5912-aaf9-4eb0-8212-fb607a65fdc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v9vc\" (UID: \"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9\") " pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:33.370669 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:33.370632 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v9vc" Jul 23 18:07:33.502789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:33.502738 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v9vc"] Jul 23 18:07:33.506534 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:33.506509 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ac5912_aaf9_4eb0_8212_fb607a65fdc9.slice/crio-c71d2ced5a9fe003e23a32f6b06befac33e4c29106a539b8bdf604f1854d028f WatchSource:0}: Error finding container c71d2ced5a9fe003e23a32f6b06befac33e4c29106a539b8bdf604f1854d028f: Status 404 returned error can't find the container with id c71d2ced5a9fe003e23a32f6b06befac33e4c29106a539b8bdf604f1854d028f Jul 23 18:07:34.460669 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:34.460622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v9vc" event={"ID":"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9","Type":"ContainerStarted","Data":"c71d2ced5a9fe003e23a32f6b06befac33e4c29106a539b8bdf604f1854d028f"} Jul 23 18:07:37.362952 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:37.362911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwjpk" Jul 23 18:07:39.479660 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:39.479579 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v9vc" event={"ID":"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9","Type":"ContainerStarted","Data":"463ebe91f4125e5d8f091525572a4a6fba3bd740a23f9d12801848bed8d8b14e"} Jul 23 18:07:41.870349 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.870301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:41.870779 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.870420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:41.873490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.873464 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 18:07:41.873650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.873500 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 18:07:41.883239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.883209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b766b43e-8eb4-43bd-9671-ed7c0c3a0e98-original-pull-secret\") pod \"global-pull-secret-syncer-bmskp\" (UID: \"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98\") " pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:41.883386 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.883311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/114cc0ef-9e50-4957-95ba-44398bf8de22-metrics-certs\") pod \"network-metrics-daemon-x2wcs\" (UID: \"114cc0ef-9e50-4957-95ba-44398bf8de22\") " pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:41.970955 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.970916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:41.973892 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.973871 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 18:07:41.984213 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.984190 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 18:07:41.993759 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:41.993718 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scbvz\" (UniqueName: \"kubernetes.io/projected/d21a8d63-8fb6-413c-bb9d-2c59b2f15c13-kube-api-access-scbvz\") pod \"network-check-target-w79t6\" (UID: \"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13\") " pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:42.055540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.055503 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bmskp" Jul 23 18:07:42.064482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.064448 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t929l\"" Jul 23 18:07:42.069554 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.069526 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bbklj\"" Jul 23 18:07:42.071910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.071798 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x2wcs" Jul 23 18:07:42.077886 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.077867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:42.456329 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.456298 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-smfl5" Jul 23 18:07:42.806538 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.806505 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w79t6"] Jul 23 18:07:42.811004 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:42.810971 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21a8d63_8fb6_413c_bb9d_2c59b2f15c13.slice/crio-94b4c80045ac1178fb6186da2a060919b2797f64964df50e833ad4325c3edf89 WatchSource:0}: Error finding container 94b4c80045ac1178fb6186da2a060919b2797f64964df50e833ad4325c3edf89: Status 404 returned error can't find the container with id 94b4c80045ac1178fb6186da2a060919b2797f64964df50e833ad4325c3edf89 Jul 23 18:07:42.828572 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.828547 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x2wcs"] Jul 23 18:07:42.829407 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:42.829382 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114cc0ef_9e50_4957_95ba_44398bf8de22.slice/crio-8f8905d789fbb098cd98c5e2b56e5727aff52d0876e1094a9f2552ac79380f29 WatchSource:0}: Error finding container 8f8905d789fbb098cd98c5e2b56e5727aff52d0876e1094a9f2552ac79380f29: Status 404 returned error can't find the container with id 8f8905d789fbb098cd98c5e2b56e5727aff52d0876e1094a9f2552ac79380f29 Jul 23 18:07:42.843987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:42.843961 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bmskp"] Jul 23 18:07:42.846786 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:42.846758 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb766b43e_8eb4_43bd_9671_ed7c0c3a0e98.slice/crio-61089dfdb83516a218a2436912731b3ee758fafa2f77cb1e6f734b1417a5eec0 WatchSource:0}: Error finding container 61089dfdb83516a218a2436912731b3ee758fafa2f77cb1e6f734b1417a5eec0: Status 404 returned error can't find the container with id 61089dfdb83516a218a2436912731b3ee758fafa2f77cb1e6f734b1417a5eec0 Jul 23 18:07:43.495930 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:43.495884 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v9vc" event={"ID":"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9","Type":"ContainerStarted","Data":"1f0400e0deba367b3abc34ab2104ffd6a50772531d774a84813666b985c5c054"} Jul 23 18:07:43.498208 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:43.498173 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bmskp" event={"ID":"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98","Type":"ContainerStarted","Data":"61089dfdb83516a218a2436912731b3ee758fafa2f77cb1e6f734b1417a5eec0"} Jul 23 18:07:43.499813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:43.499780 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x2wcs" event={"ID":"114cc0ef-9e50-4957-95ba-44398bf8de22","Type":"ContainerStarted","Data":"8f8905d789fbb098cd98c5e2b56e5727aff52d0876e1094a9f2552ac79380f29"} Jul 23 18:07:43.502007 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:43.501979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w79t6" event={"ID":"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13","Type":"ContainerStarted","Data":"94b4c80045ac1178fb6186da2a060919b2797f64964df50e833ad4325c3edf89"} Jul 23 18:07:44.508704 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:44.508613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x2wcs" event={"ID":"114cc0ef-9e50-4957-95ba-44398bf8de22","Type":"ContainerStarted","Data":"07e4887b917cde6033a1ccf7aa6b6fa65b418c0a826f21380622ca986a388703"} Jul 23 18:07:44.508704 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:44.508688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x2wcs" event={"ID":"114cc0ef-9e50-4957-95ba-44398bf8de22","Type":"ContainerStarted","Data":"2e6139f439b10f89b1fed9112ca8067e219a081c332e5d89729f634d4c09d9f1"} Jul 23 18:07:44.534520 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:44.534354 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x2wcs" podStartSLOduration=66.285275755 podStartE2EDuration="1m7.534336565s" totalImagesPullingTime="1.24906081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:44.533563568 +0000 UTC m=+68.101160739" watchObservedRunningTime="2026-07-23 18:07:44.534336565 +0000 UTC m=+68.101933752" Jul 23 18:07:49.455815 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.455777 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:07:49.461009 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.460976 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.464291 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.464269 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 18:07:49.465129 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465105 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 18:07:49.465393 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465368 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f6wlv\"" Jul 23 18:07:49.465496 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465424 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 18:07:49.465496 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 18:07:49.465662 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465632 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 18:07:49.465662 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.465646 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 18:07:49.466032 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.466012 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 18:07:49.474127 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.474080 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 18:07:49.475255 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.475221 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:07:49.528830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.528802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.528830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.528836 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.529068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.528865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.529068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.528904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.529068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.528982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.529068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.529022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.529068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.529062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrtp\" (UniqueName: \"kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.629907 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.629864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.629919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.629956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.629982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630080 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrtp\" (UniqueName: \"kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630824 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630796 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.630949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.630910 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.631115 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.631090 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.632910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.632873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.632910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.632876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.641945 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.641907 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrtp\" (UniqueName: \"kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp\") pod \"console-6f69f5695b-blllv\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.771980 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.771888 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:49.984543 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:49.984480 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:07:49.986506 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:49.986472 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c76d89d_a5e9_45d7_97bc_a24937484d09.slice/crio-f59e0ed23597a11e8de1012697d7f4676810e48c13888ce2d5b61664e6e52204 WatchSource:0}: Error finding container f59e0ed23597a11e8de1012697d7f4676810e48c13888ce2d5b61664e6e52204: Status 404 returned error can't find the container with id f59e0ed23597a11e8de1012697d7f4676810e48c13888ce2d5b61664e6e52204 Jul 23 18:07:50.532914 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.532876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v9vc" event={"ID":"f1ac5912-aaf9-4eb0-8212-fb607a65fdc9","Type":"ContainerStarted","Data":"6ade2bb403f3c50cb19ae641fefef022c0c2727b7da4c48048f198133d985b95"} Jul 23 18:07:50.534250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.534226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bmskp" event={"ID":"b766b43e-8eb4-43bd-9671-ed7c0c3a0e98","Type":"ContainerStarted","Data":"ebf75d3551e45fe9523c036fb9199af2370a30c7de4c4f0f8223fc0930410440"} Jul 23 18:07:50.535478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.535459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w79t6" event={"ID":"d21a8d63-8fb6-413c-bb9d-2c59b2f15c13","Type":"ContainerStarted","Data":"9690365de064e42d10072d339abf9450d89e7b5d29618af58c1e85ee07f1e119"} Jul 23 18:07:50.535566 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.535502 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:07:50.536507 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.536489 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f69f5695b-blllv" event={"ID":"7c76d89d-a5e9-45d7-97bc-a24937484d09","Type":"ContainerStarted","Data":"f59e0ed23597a11e8de1012697d7f4676810e48c13888ce2d5b61664e6e52204"} Jul 23 18:07:50.555650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.555606 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5v9vc" podStartSLOduration=17.363407557 podStartE2EDuration="33.555592455s" totalImagesPullingTime="16.192184898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:07:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:50.554378288 +0000 UTC m=+74.121975459" watchObservedRunningTime="2026-07-23 18:07:50.555592455 +0000 UTC m=+74.123189626" Jul 23 18:07:50.574657 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:50.574602 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bmskp" podStartSLOduration=66.575884908 podStartE2EDuration="1m13.574581243s" totalImagesPullingTime="6.998696335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:50.574351688 +0000 UTC m=+74.141948877" watchObservedRunningTime="2026-07-23 18:07:50.574581243 +0000 UTC m=+74.142178416" Jul 23 18:07:53.549821 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:53.549782 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f69f5695b-blllv" event={"ID":"7c76d89d-a5e9-45d7-97bc-a24937484d09","Type":"ContainerStarted","Data":"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a"} Jul 23 18:07:53.577180 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:53.577116 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f69f5695b-blllv" podStartSLOduration=1.6168718439999998 podStartE2EDuration="4.577096374s" totalImagesPullingTime="2.96022453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:53.576896031 +0000 UTC m=+77.144493216" watchObservedRunningTime="2026-07-23 18:07:53.577096374 +0000 UTC m=+77.144693543" Jul 23 18:07:53.577632 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:53.577601 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w79t6" podStartSLOduration=69.543633682 podStartE2EDuration="1m16.577589882s" totalImagesPullingTime="7.0339562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:07:50.591993099 +0000 UTC m=+74.159590269" watchObservedRunningTime="2026-07-23 18:07:53.577589882 +0000 UTC m=+77.145187058" Jul 23 18:07:57.468467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.468429 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh"] Jul 23 18:07:57.470404 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.470388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.473302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.473281 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 18:07:57.473441 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.473346 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 18:07:57.473441 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.473407 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cptvd\"" Jul 23 18:07:57.474436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.474423 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 18:07:57.474497 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.474461 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 18:07:57.474538 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.474516 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 18:07:57.492312 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.492273 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh"] Jul 23 18:07:57.522239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.522199 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5zbbx"] Jul 23 18:07:57.524379 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.524354 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.526893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.526865 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 18:07:57.527004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.526889 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 18:07:57.527004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.526901 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mjcbz\"" Jul 23 18:07:57.527004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.526902 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 18:07:57.592173 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vb24\" (UniqueName: \"kubernetes.io/projected/baa81db5-e393-4625-b648-cd2e5da1566d-kube-api-access-6vb24\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592173 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-sys\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-wtmp\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592265 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228f96e-2f05-451d-8230-0c11be457cb4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.592377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-root\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592449 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592477 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-textfile\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvtd\" (UniqueName: \"kubernetes.io/projected/b228f96e-2f05-451d-8230-0c11be457cb4-kube-api-access-cxvtd\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.592528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.592519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-metrics-client-ca\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693208 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.693208 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-textfile\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693236 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvtd\" (UniqueName: \"kubernetes.io/projected/b228f96e-2f05-451d-8230-0c11be457cb4-kube-api-access-cxvtd\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.693429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-metrics-client-ca\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vb24\" (UniqueName: \"kubernetes.io/projected/baa81db5-e393-4625-b648-cd2e5da1566d-kube-api-access-6vb24\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-sys\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693429 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693409 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-wtmp\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693506 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228f96e-2f05-451d-8230-0c11be457cb4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-sys\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-root\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.693658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-wtmp\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa81db5-e393-4625-b648-cd2e5da1566d-root\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693677 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-textfile\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:57.693798 2570 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:57.693804 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:57.693870 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls podName:baa81db5-e393-4625-b648-cd2e5da1566d nodeName:}" failed. No retries permitted until 2026-07-23 18:07:58.193850795 +0000 UTC m=+81.761447957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls") pod "node-exporter-5zbbx" (UID: "baa81db5-e393-4625-b648-cd2e5da1566d") : secret "node-exporter-tls" not found Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:57.693888 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls podName:b228f96e-2f05-451d-8230-0c11be457cb4 nodeName:}" failed. No retries permitted until 2026-07-23 18:07:58.193878006 +0000 UTC m=+81.761475162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-5tddh" (UID: "b228f96e-2f05-451d-8230-0c11be457cb4") : secret "openshift-state-metrics-tls" not found Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.693928 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-metrics-client-ca\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.694064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.694037 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.694330 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.694240 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b228f96e-2f05-451d-8230-0c11be457cb4-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.695687 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.695666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:57.695805 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.695785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.703317 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.703288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vb24\" (UniqueName: \"kubernetes.io/projected/baa81db5-e393-4625-b648-cd2e5da1566d-kube-api-access-6vb24\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:57.703317 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:57.703304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvtd\" (UniqueName: \"kubernetes.io/projected/b228f96e-2f05-451d-8230-0c11be457cb4-kube-api-access-cxvtd\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:58.197903 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.197775 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:58.197903 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.197828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:58.198101 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:58.197938 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 18:07:58.198101 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:07:58.198028 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls podName:baa81db5-e393-4625-b648-cd2e5da1566d nodeName:}" failed. No retries permitted until 2026-07-23 18:07:59.198007273 +0000 UTC m=+82.765604456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls") pod "node-exporter-5zbbx" (UID: "baa81db5-e393-4625-b648-cd2e5da1566d") : secret "node-exporter-tls" not found Jul 23 18:07:58.200481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.200454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b228f96e-2f05-451d-8230-0c11be457cb4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-5tddh\" (UID: \"b228f96e-2f05-451d-8230-0c11be457cb4\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:58.379392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.379352 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" Jul 23 18:07:58.517697 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.517662 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh"] Jul 23 18:07:58.521188 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:58.521166 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb228f96e_2f05_451d_8230_0c11be457cb4.slice/crio-0a9bb9d05783cefb831176b21efa11ff8dee053b6641f5bc3ed33eade0ec623a WatchSource:0}: Error finding container 0a9bb9d05783cefb831176b21efa11ff8dee053b6641f5bc3ed33eade0ec623a: Status 404 returned error can't find the container with id 0a9bb9d05783cefb831176b21efa11ff8dee053b6641f5bc3ed33eade0ec623a Jul 23 18:07:58.565665 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:58.565637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" event={"ID":"b228f96e-2f05-451d-8230-0c11be457cb4","Type":"ContainerStarted","Data":"0a9bb9d05783cefb831176b21efa11ff8dee053b6641f5bc3ed33eade0ec623a"} Jul 23 18:07:59.207063 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.207025 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:59.209374 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.209348 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa81db5-e393-4625-b648-cd2e5da1566d-node-exporter-tls\") pod \"node-exporter-5zbbx\" (UID: \"baa81db5-e393-4625-b648-cd2e5da1566d\") " pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:59.333433 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.333389 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zbbx" Jul 23 18:07:59.343791 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:07:59.343758 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa81db5_e393_4625_b648_cd2e5da1566d.slice/crio-a02ea3797b723ede382aa17f81011771456f3998d2cb6b4f20a3b6134632921a WatchSource:0}: Error finding container a02ea3797b723ede382aa17f81011771456f3998d2cb6b4f20a3b6134632921a: Status 404 returned error can't find the container with id a02ea3797b723ede382aa17f81011771456f3998d2cb6b4f20a3b6134632921a Jul 23 18:07:59.572355 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.572270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" event={"ID":"b228f96e-2f05-451d-8230-0c11be457cb4","Type":"ContainerStarted","Data":"a531fd3dc80446486ad399ca2165b66965176ba6489d494416b265f90fe96b4b"} Jul 23 18:07:59.572355 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.572340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" event={"ID":"b228f96e-2f05-451d-8230-0c11be457cb4","Type":"ContainerStarted","Data":"0b7ac7d8ef130af8a1714bc8746c0a6cd8d0f872690cde2972e74bbf13819de0"} Jul 23 18:07:59.573633 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.573588 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zbbx" event={"ID":"baa81db5-e393-4625-b648-cd2e5da1566d","Type":"ContainerStarted","Data":"a02ea3797b723ede382aa17f81011771456f3998d2cb6b4f20a3b6134632921a"} Jul 23 18:07:59.772636 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.772596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:59.772771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.772660 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:07:59.777385 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:07:59.777361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:08:00.578809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:00.578767 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" event={"ID":"b228f96e-2f05-451d-8230-0c11be457cb4","Type":"ContainerStarted","Data":"b556b7c8bf789d6a1cbb685ca71c2e4cdafb38b5deaa2f0d892dbae553f3bd71"} Jul 23 18:08:00.580353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:00.580330 2570 generic.go:356] "Generic (PLEG): container finished" podID="baa81db5-e393-4625-b648-cd2e5da1566d" containerID="9a03381c183e9cd2f1eecafa22f7b3e88bfa2a04302fab0343e7ab25c73944b9" exitCode=0 Jul 23 18:08:00.580452 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:00.580418 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zbbx" event={"ID":"baa81db5-e393-4625-b648-cd2e5da1566d","Type":"ContainerDied","Data":"9a03381c183e9cd2f1eecafa22f7b3e88bfa2a04302fab0343e7ab25c73944b9"} Jul 23 18:08:00.585053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:00.585029 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:08:00.618121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:00.618063 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-5tddh" podStartSLOduration=2.502142466 podStartE2EDuration="3.618047284s" totalImagesPullingTime="1.115904818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:00.616025175 +0000 UTC m=+84.183622373" watchObservedRunningTime="2026-07-23 18:08:00.618047284 +0000 UTC m=+84.185644454" Jul 23 18:08:01.585733 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:01.585687 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zbbx" event={"ID":"baa81db5-e393-4625-b648-cd2e5da1566d","Type":"ContainerStarted","Data":"a5af114371cac7bbaede4929300bc54f42f74777ba5567a93cbd8a1aec442a5b"} Jul 23 18:08:01.586106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:01.585778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zbbx" event={"ID":"baa81db5-e393-4625-b648-cd2e5da1566d","Type":"ContainerStarted","Data":"4b2f890c53813299ea4bde2f802ba86f9f852e309494049cebe896b3ef546655"} Jul 23 18:08:08.555464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.555391 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5zbbx" podStartSLOduration=10.576743343 podStartE2EDuration="11.555373015s" totalImagesPullingTime="912.171192ms" totalInitContainerRuntime="66.45848ms" isStatefulPod=true podCreationTimestamp="2026-07-23 18:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:01.610781527 +0000 UTC m=+85.178378697" watchObservedRunningTime="2026-07-23 18:08:08.555373015 +0000 UTC m=+92.122970185" Jul 23 18:08:08.556095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.556076 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:08:08.599405 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.599369 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:08:08.599405 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.599390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678514 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678469 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678687 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqfx\" (UniqueName: \"kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.678851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.678733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.779897 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.779897 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqfx\" (UniqueName: \"kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.779993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.780022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780762 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.780713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.780884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.780724 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.781062 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.781034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.781263 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.781241 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.782446 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.782424 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.782657 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.782637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.795853 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.795820 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqfx\" (UniqueName: \"kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx\") pod \"console-6d7f6fb568-5xlms\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:08.909729 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:08.909676 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:09.099134 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:09.099101 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:08:09.102620 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:08:09.102580 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04235db7_c450_4b68_9b74_90fbc3b62475.slice/crio-8f2ca4e8a5e0a1a203c2716151c623a8ffae8b846ba4204040cfc74b8c18709c WatchSource:0}: Error finding container 8f2ca4e8a5e0a1a203c2716151c623a8ffae8b846ba4204040cfc74b8c18709c: Status 404 returned error can't find the container with id 8f2ca4e8a5e0a1a203c2716151c623a8ffae8b846ba4204040cfc74b8c18709c Jul 23 18:08:09.613505 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:09.613473 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7f6fb568-5xlms" event={"ID":"04235db7-c450-4b68-9b74-90fbc3b62475","Type":"ContainerStarted","Data":"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636"} Jul 23 18:08:09.613908 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:09.613525 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7f6fb568-5xlms" event={"ID":"04235db7-c450-4b68-9b74-90fbc3b62475","Type":"ContainerStarted","Data":"8f2ca4e8a5e0a1a203c2716151c623a8ffae8b846ba4204040cfc74b8c18709c"} Jul 23 18:08:09.643884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:09.643829 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d7f6fb568-5xlms" podStartSLOduration=1.6438097 podStartE2EDuration="1.6438097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:08:09.641776791 +0000 UTC m=+93.209373979" watchObservedRunningTime="2026-07-23 18:08:09.6438097 +0000 UTC m=+93.211406870" Jul 23 18:08:18.909826 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:18.909786 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:18.910265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:18.909881 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:18.915167 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:18.915144 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:19.651949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:19.651922 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:08:19.745621 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:19.745591 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:08:21.543883 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:21.543848 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w79t6" Jul 23 18:08:44.766202 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:44.766140 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f69f5695b-blllv" podUID="7c76d89d-a5e9-45d7-97bc-a24937484d09" containerName="console" containerID="cri-o://db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a" gracePeriod=15 Jul 23 18:08:45.004459 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.004432 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:08:45.032551 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.032458 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" podUID="f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 18:08:45.159589 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159553 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159606 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159630 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159651 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whrtp\" (UniqueName: \"kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159703 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.159781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.159736 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config\") pod \"7c76d89d-a5e9-45d7-97bc-a24937484d09\" (UID: \"7c76d89d-a5e9-45d7-97bc-a24937484d09\") " Jul 23 18:08:45.160190 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.160110 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:45.160190 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.160120 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config" (OuterVolumeSpecName: "console-config") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:45.160321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.160247 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:45.160321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.160296 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:08:45.162211 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.162180 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:08:45.162377 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.162358 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:08:45.162431 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.162380 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp" (OuterVolumeSpecName: "kube-api-access-whrtp") pod "7c76d89d-a5e9-45d7-97bc-a24937484d09" (UID: "7c76d89d-a5e9-45d7-97bc-a24937484d09"). InnerVolumeSpecName "kube-api-access-whrtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:08:45.260392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260355 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-oauth-serving-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260387 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-config\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260397 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-serving-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260405 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrtp\" (UniqueName: \"kubernetes.io/projected/7c76d89d-a5e9-45d7-97bc-a24937484d09-kube-api-access-whrtp\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260414 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260422 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c76d89d-a5e9-45d7-97bc-a24937484d09-trusted-ca-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.260643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.260430 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c76d89d-a5e9-45d7-97bc-a24937484d09-console-oauth-config\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:08:45.739840 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.739806 2570 generic.go:356] "Generic (PLEG): container finished" podID="7c76d89d-a5e9-45d7-97bc-a24937484d09" containerID="db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a" exitCode=0 Jul 23 18:08:45.740003 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.739853 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f69f5695b-blllv" Jul 23 18:08:45.740003 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.739898 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f69f5695b-blllv" event={"ID":"7c76d89d-a5e9-45d7-97bc-a24937484d09","Type":"ContainerDied","Data":"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a"} Jul 23 18:08:45.740003 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.739950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f69f5695b-blllv" event={"ID":"7c76d89d-a5e9-45d7-97bc-a24937484d09","Type":"ContainerDied","Data":"f59e0ed23597a11e8de1012697d7f4676810e48c13888ce2d5b61664e6e52204"} Jul 23 18:08:45.740003 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.739975 2570 scope.go:117] "RemoveContainer" containerID="db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a" Jul 23 18:08:45.748788 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.748773 2570 scope.go:117] "RemoveContainer" containerID="db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a" Jul 23 18:08:45.749047 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:08:45.749025 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a\": container with ID starting with db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a not found: ID does not exist" containerID="db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a" Jul 23 18:08:45.749097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.749055 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a"} err="failed to get container status \"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a\": rpc error: code = NotFound desc = could not find container \"db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a\": container with ID starting with db3bdd44d37edd80213cc121ebeab71cfec1716176fda49e5d6b7c1748b32e0a not found: ID does not exist" Jul 23 18:08:45.773814 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.773783 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:08:45.776422 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:45.776400 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f69f5695b-blllv"] Jul 23 18:08:47.141205 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:47.141167 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c76d89d-a5e9-45d7-97bc-a24937484d09" path="/var/lib/kubelet/pods/7c76d89d-a5e9-45d7-97bc-a24937484d09/volumes" Jul 23 18:08:55.032825 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:08:55.032783 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" podUID="f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 18:09:05.032656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.032614 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" podUID="f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 23 18:09:05.033045 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.032712 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" Jul 23 18:09:05.033222 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.033197 2570 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ba7b776d34a9cdd758b2430e333570a285e407f9d3a2be6a6ff27d6247e3ac08"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 23 18:09:05.033258 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.033243 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" podUID="f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9" containerName="service-proxy" containerID="cri-o://ba7b776d34a9cdd758b2430e333570a285e407f9d3a2be6a6ff27d6247e3ac08" gracePeriod=30 Jul 23 18:09:05.813604 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.813565 2570 generic.go:356] "Generic (PLEG): container finished" podID="f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9" containerID="ba7b776d34a9cdd758b2430e333570a285e407f9d3a2be6a6ff27d6247e3ac08" exitCode=2 Jul 23 18:09:05.813806 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.813623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerDied","Data":"ba7b776d34a9cdd758b2430e333570a285e407f9d3a2be6a6ff27d6247e3ac08"} Jul 23 18:09:05.813806 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:05.813661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69584f9df7-phjxc" event={"ID":"f335e3c1-3fcc-4af2-82dc-8f6aeccf92a9","Type":"ContainerStarted","Data":"d5507391f42ac214a3583008224e1e4951bf3e1f374a30661a1ba64f76fc1b20"} Jul 23 18:09:31.119996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:31.119917 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:09:56.143906 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.143852 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d7f6fb568-5xlms" podUID="04235db7-c450-4b68-9b74-90fbc3b62475" containerName="console" containerID="cri-o://2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636" gracePeriod=15 Jul 23 18:09:56.391511 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.391482 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:09:56.457066 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.456979 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457066 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457079 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457134 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njqfx\" (UniqueName: \"kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457442 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457290 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457442 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457347 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config\") pod \"04235db7-c450-4b68-9b74-90fbc3b62475\" (UID: \"04235db7-c450-4b68-9b74-90fbc3b62475\") " Jul 23 18:09:56.457642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:09:56.457642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca" (OuterVolumeSpecName: "service-ca") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:09:56.457863 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457717 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:09:56.457863 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.457853 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config" (OuterVolumeSpecName: "console-config") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:09:56.459333 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.459305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:09:56.459428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.459347 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx" (OuterVolumeSpecName: "kube-api-access-njqfx") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "kube-api-access-njqfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:09:56.459428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.459350 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04235db7-c450-4b68-9b74-90fbc3b62475" (UID: "04235db7-c450-4b68-9b74-90fbc3b62475"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:09:56.557928 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557890 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-trusted-ca-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.557928 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557922 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.557928 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557932 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-oauth-config\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.558161 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557941 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqfx\" (UniqueName: \"kubernetes.io/projected/04235db7-c450-4b68-9b74-90fbc3b62475-kube-api-access-njqfx\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.558161 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557951 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-oauth-serving-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.558161 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557959 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04235db7-c450-4b68-9b74-90fbc3b62475-console-config\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.558161 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.557968 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04235db7-c450-4b68-9b74-90fbc3b62475-console-serving-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:09:56.983564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.983533 2570 generic.go:356] "Generic (PLEG): container finished" podID="04235db7-c450-4b68-9b74-90fbc3b62475" containerID="2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636" exitCode=0 Jul 23 18:09:56.983791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.983580 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d7f6fb568-5xlms" Jul 23 18:09:56.983791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.983623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7f6fb568-5xlms" event={"ID":"04235db7-c450-4b68-9b74-90fbc3b62475","Type":"ContainerDied","Data":"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636"} Jul 23 18:09:56.983791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.983713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d7f6fb568-5xlms" event={"ID":"04235db7-c450-4b68-9b74-90fbc3b62475","Type":"ContainerDied","Data":"8f2ca4e8a5e0a1a203c2716151c623a8ffae8b846ba4204040cfc74b8c18709c"} Jul 23 18:09:56.983791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.983766 2570 scope.go:117] "RemoveContainer" containerID="2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636" Jul 23 18:09:56.992636 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.992618 2570 scope.go:117] "RemoveContainer" containerID="2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636" Jul 23 18:09:56.992977 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:09:56.992951 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636\": container with ID starting with 2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636 not found: ID does not exist" containerID="2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636" Jul 23 18:09:56.993034 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:56.992986 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636"} err="failed to get container status \"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636\": rpc error: code = NotFound desc = could not find container \"2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636\": container with ID starting with 2771f7a1e0ef44dbe78c54b5a4158e59e7fd04f8f832e1a8283d2899d4458636 not found: ID does not exist" Jul 23 18:09:57.007312 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:57.007283 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:09:57.012919 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:57.012895 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d7f6fb568-5xlms"] Jul 23 18:09:57.140763 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:09:57.140698 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04235db7-c450-4b68-9b74-90fbc3b62475" path="/var/lib/kubelet/pods/04235db7-c450-4b68-9b74-90fbc3b62475/volumes" Jul 23 18:10:27.089272 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089237 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl"] Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089473 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c76d89d-a5e9-45d7-97bc-a24937484d09" containerName="console" Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089483 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c76d89d-a5e9-45d7-97bc-a24937484d09" containerName="console" Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089492 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04235db7-c450-4b68-9b74-90fbc3b62475" containerName="console" Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089497 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="04235db7-c450-4b68-9b74-90fbc3b62475" containerName="console" Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089535 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="04235db7-c450-4b68-9b74-90fbc3b62475" containerName="console" Jul 23 18:10:27.089716 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.089544 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c76d89d-a5e9-45d7-97bc-a24937484d09" containerName="console" Jul 23 18:10:27.091837 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.091820 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.095252 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.095234 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:10:27.096443 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.096428 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-98wls\"" Jul 23 18:10:27.096506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.096434 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:10:27.116336 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.116298 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl"] Jul 23 18:10:27.269989 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.269955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.269989 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.269992 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.270242 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.270031 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xnm\" (UniqueName: \"kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.371001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.370914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xnm\" (UniqueName: \"kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.371001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.370985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.371206 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.371016 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.371355 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.371333 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.371403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.371385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.382411 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.382371 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xnm\" (UniqueName: \"kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.400369 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.400338 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:27.542314 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:27.542276 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl"] Jul 23 18:10:27.546040 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:10:27.546011 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode845fdea_508b_43d7_966f_0c1437a4799d.slice/crio-27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030 WatchSource:0}: Error finding container 27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030: Status 404 returned error can't find the container with id 27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030 Jul 23 18:10:28.090546 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:28.090497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" event={"ID":"e845fdea-508b-43d7-966f-0c1437a4799d","Type":"ContainerStarted","Data":"27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030"} Jul 23 18:10:35.116719 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:35.116680 2570 generic.go:356] "Generic (PLEG): container finished" podID="e845fdea-508b-43d7-966f-0c1437a4799d" containerID="e223db086563f7ef8ecc91c75b2d10854c89dcb977541025cf1614609f8e11ad" exitCode=0 Jul 23 18:10:35.117198 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:35.116769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" event={"ID":"e845fdea-508b-43d7-966f-0c1437a4799d","Type":"ContainerDied","Data":"e223db086563f7ef8ecc91c75b2d10854c89dcb977541025cf1614609f8e11ad"} Jul 23 18:10:38.129156 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:38.129106 2570 generic.go:356] "Generic (PLEG): container finished" podID="e845fdea-508b-43d7-966f-0c1437a4799d" containerID="ffb4307c28fd818d920b524163b0b57d34061db4cf4b3e42ec8e940ddf4cbd93" exitCode=0 Jul 23 18:10:38.129524 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:38.129174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" event={"ID":"e845fdea-508b-43d7-966f-0c1437a4799d","Type":"ContainerDied","Data":"ffb4307c28fd818d920b524163b0b57d34061db4cf4b3e42ec8e940ddf4cbd93"} Jul 23 18:10:46.160233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:46.160137 2570 generic.go:356] "Generic (PLEG): container finished" podID="e845fdea-508b-43d7-966f-0c1437a4799d" containerID="b83e1e7031fcf376fb8e54fe81c88c5b581dcb61001604520a89b1ee235ea5b4" exitCode=0 Jul 23 18:10:46.160233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:46.160187 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" event={"ID":"e845fdea-508b-43d7-966f-0c1437a4799d","Type":"ContainerDied","Data":"b83e1e7031fcf376fb8e54fe81c88c5b581dcb61001604520a89b1ee235ea5b4"} Jul 23 18:10:47.290356 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.290327 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:47.326955 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.326925 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util\") pod \"e845fdea-508b-43d7-966f-0c1437a4799d\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " Jul 23 18:10:47.327133 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.326985 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95xnm\" (UniqueName: \"kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm\") pod \"e845fdea-508b-43d7-966f-0c1437a4799d\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " Jul 23 18:10:47.327133 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.327011 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle\") pod \"e845fdea-508b-43d7-966f-0c1437a4799d\" (UID: \"e845fdea-508b-43d7-966f-0c1437a4799d\") " Jul 23 18:10:47.327653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.327613 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle" (OuterVolumeSpecName: "bundle") pod "e845fdea-508b-43d7-966f-0c1437a4799d" (UID: "e845fdea-508b-43d7-966f-0c1437a4799d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:10:47.329351 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.329319 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm" (OuterVolumeSpecName: "kube-api-access-95xnm") pod "e845fdea-508b-43d7-966f-0c1437a4799d" (UID: "e845fdea-508b-43d7-966f-0c1437a4799d"). InnerVolumeSpecName "kube-api-access-95xnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:10:47.331321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.331295 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util" (OuterVolumeSpecName: "util") pod "e845fdea-508b-43d7-966f-0c1437a4799d" (UID: "e845fdea-508b-43d7-966f-0c1437a4799d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:10:47.427604 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.427512 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:10:47.427604 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.427546 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e845fdea-508b-43d7-966f-0c1437a4799d-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:10:47.427604 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:47.427556 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xnm\" (UniqueName: \"kubernetes.io/projected/e845fdea-508b-43d7-966f-0c1437a4799d-kube-api-access-95xnm\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:10:48.168661 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:48.168626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" event={"ID":"e845fdea-508b-43d7-966f-0c1437a4799d","Type":"ContainerDied","Data":"27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030"} Jul 23 18:10:48.168661 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:48.168645 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kjvjl" Jul 23 18:10:48.168893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:48.168676 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27000327101eb2cc9be0b79fa48938fac813c5a423fccff04a33ae8742228030" Jul 23 18:10:55.357441 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357388 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv"] Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357634 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="extract" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357646 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="extract" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357657 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="util" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357662 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="util" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357674 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="pull" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357683 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="pull" Jul 23 18:10:55.357809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.357730 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e845fdea-508b-43d7-966f-0c1437a4799d" containerName="extract" Jul 23 18:10:55.361023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.360998 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.363887 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.363865 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:10:55.364019 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.363927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 18:10:55.364019 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.363998 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zs8gn\"" Jul 23 18:10:55.378105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.378077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv"] Jul 23 18:10:55.482419 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.482375 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lwm\" (UniqueName: \"kubernetes.io/projected/3537b557-72f8-48f3-94e6-87595ba9511a-kube-api-access-z8lwm\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.482623 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.482509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3537b557-72f8-48f3-94e6-87595ba9511a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.583302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.583246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3537b557-72f8-48f3-94e6-87595ba9511a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.583475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.583319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lwm\" (UniqueName: \"kubernetes.io/projected/3537b557-72f8-48f3-94e6-87595ba9511a-kube-api-access-z8lwm\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.583703 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.583678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3537b557-72f8-48f3-94e6-87595ba9511a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.593969 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.593932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lwm\" (UniqueName: \"kubernetes.io/projected/3537b557-72f8-48f3-94e6-87595ba9511a-kube-api-access-z8lwm\") pod \"cert-manager-operator-controller-manager-55c444bf86-sxnqv\" (UID: \"3537b557-72f8-48f3-94e6-87595ba9511a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.669911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.669821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" Jul 23 18:10:55.812360 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:55.812323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv"] Jul 23 18:10:55.815503 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:10:55.815467 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3537b557_72f8_48f3_94e6_87595ba9511a.slice/crio-cae850d8747a3ceb6d62a9b2d8c7ac493009430f5de29382d9d1e5a173a8de5d WatchSource:0}: Error finding container cae850d8747a3ceb6d62a9b2d8c7ac493009430f5de29382d9d1e5a173a8de5d: Status 404 returned error can't find the container with id cae850d8747a3ceb6d62a9b2d8c7ac493009430f5de29382d9d1e5a173a8de5d Jul 23 18:10:56.198384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:56.198342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" event={"ID":"3537b557-72f8-48f3-94e6-87595ba9511a","Type":"ContainerStarted","Data":"cae850d8747a3ceb6d62a9b2d8c7ac493009430f5de29382d9d1e5a173a8de5d"} Jul 23 18:10:58.208174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:58.208099 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" event={"ID":"3537b557-72f8-48f3-94e6-87595ba9511a","Type":"ContainerStarted","Data":"3a189eedab92c19f0f4ae37416628119e0467ed6ac1579d9a4c0826415addeb4"} Jul 23 18:10:58.273585 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:58.273525 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sxnqv" podStartSLOduration=1.804125067 podStartE2EDuration="3.273507371s" totalImagesPullingTime="1.469382304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:10:58.269860382 +0000 UTC m=+261.837457579" watchObservedRunningTime="2026-07-23 18:10:58.273507371 +0000 UTC m=+261.841104542" Jul 23 18:10:58.987492 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:58.987458 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl"] Jul 23 18:10:58.989842 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:58.989816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.007008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.006979 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:10:59.007199 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.007147 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-98wls\"" Jul 23 18:10:59.007262 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:10:59.007182 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" is forbidden: User \"system:node:ip-10-0-132-197.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-132-197.ec2.internal' and this object" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.008291 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.008274 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:10:59.033276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.033244 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl"] Jul 23 18:10:59.112688 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.112655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.112861 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.112695 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdp5\" (UniqueName: \"kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.112861 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.112732 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.213348 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.213319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.213348 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.213357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdp5\" (UniqueName: \"kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.213775 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.213383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.213775 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.213713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.213843 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.213787 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.235095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.235062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdp5\" (UniqueName: \"kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.299126 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.299038 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:10:59.445785 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:10:59.445734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl"] Jul 23 18:10:59.448616 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:10:59.448585 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d2e922_3d44_4e47_a482_72a6eb248b91.slice/crio-6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3 WatchSource:0}: Error finding container 6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3: Status 404 returned error can't find the container with id 6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3 Jul 23 18:11:00.216849 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:00.216815 2570 generic.go:356] "Generic (PLEG): container finished" podID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerID="12ba20287414495e98a4b1037a71e1da80669a0dc5969f4d1b9cef75486f474a" exitCode=0 Jul 23 18:11:00.217286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:00.216897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" event={"ID":"03d2e922-3d44-4e47-a482-72a6eb248b91","Type":"ContainerDied","Data":"12ba20287414495e98a4b1037a71e1da80669a0dc5969f4d1b9cef75486f474a"} Jul 23 18:11:00.217286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:00.216951 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" event={"ID":"03d2e922-3d44-4e47-a482-72a6eb248b91","Type":"ContainerStarted","Data":"6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3"} Jul 23 18:11:03.234114 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:03.234077 2570 generic.go:356] "Generic (PLEG): container finished" podID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerID="3335ed0a1d200aecbb2b78610b9e77bc3a08ece6c623686f55e445523beaa283" exitCode=0 Jul 23 18:11:03.234511 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:03.234130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" event={"ID":"03d2e922-3d44-4e47-a482-72a6eb248b91","Type":"ContainerDied","Data":"3335ed0a1d200aecbb2b78610b9e77bc3a08ece6c623686f55e445523beaa283"} Jul 23 18:11:04.240145 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:04.240113 2570 generic.go:356] "Generic (PLEG): container finished" podID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerID="f255948dd378687dc0d266a45be31d66243ce6cac144d854f727c477cda9a09f" exitCode=0 Jul 23 18:11:04.240580 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:04.240185 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" event={"ID":"03d2e922-3d44-4e47-a482-72a6eb248b91","Type":"ContainerDied","Data":"f255948dd378687dc0d266a45be31d66243ce6cac144d854f727c477cda9a09f"} Jul 23 18:11:05.368850 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.368822 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:11:05.462970 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.462929 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrdp5\" (UniqueName: \"kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5\") pod \"03d2e922-3d44-4e47-a482-72a6eb248b91\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " Jul 23 18:11:05.462970 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.462978 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util\") pod \"03d2e922-3d44-4e47-a482-72a6eb248b91\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " Jul 23 18:11:05.463160 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.463008 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle\") pod \"03d2e922-3d44-4e47-a482-72a6eb248b91\" (UID: \"03d2e922-3d44-4e47-a482-72a6eb248b91\") " Jul 23 18:11:05.463416 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.463397 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle" (OuterVolumeSpecName: "bundle") pod "03d2e922-3d44-4e47-a482-72a6eb248b91" (UID: "03d2e922-3d44-4e47-a482-72a6eb248b91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:11:05.465240 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.465202 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5" (OuterVolumeSpecName: "kube-api-access-xrdp5") pod "03d2e922-3d44-4e47-a482-72a6eb248b91" (UID: "03d2e922-3d44-4e47-a482-72a6eb248b91"). InnerVolumeSpecName "kube-api-access-xrdp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:11:05.467925 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.467900 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util" (OuterVolumeSpecName: "util") pod "03d2e922-3d44-4e47-a482-72a6eb248b91" (UID: "03d2e922-3d44-4e47-a482-72a6eb248b91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:11:05.564351 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.564238 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:05.564351 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.564288 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdp5\" (UniqueName: \"kubernetes.io/projected/03d2e922-3d44-4e47-a482-72a6eb248b91-kube-api-access-xrdp5\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:05.564351 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.564301 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03d2e922-3d44-4e47-a482-72a6eb248b91-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:05.902627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902584 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm"] Jul 23 18:11:05.902935 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902917 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="pull" Jul 23 18:11:05.902935 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902937 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="pull" Jul 23 18:11:05.903095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902965 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="util" Jul 23 18:11:05.903095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902974 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="util" Jul 23 18:11:05.903095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902985 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="extract" Jul 23 18:11:05.903095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.902996 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="extract" Jul 23 18:11:05.903095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.903071 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d2e922-3d44-4e47-a482-72a6eb248b91" containerName="extract" Jul 23 18:11:05.905113 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.905091 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:05.908547 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.908522 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 18:11:05.909816 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.909798 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ww49x\"" Jul 23 18:11:05.909928 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.909866 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 18:11:05.917174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.917149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm"] Jul 23 18:11:05.967206 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.967163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjn5s\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-kube-api-access-qjn5s\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:05.967384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:05.967224 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.068383 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.068336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjn5s\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-kube-api-access-qjn5s\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.068585 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.068408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.082253 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.082211 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.082448 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.082427 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjn5s\" (UniqueName: \"kubernetes.io/projected/b1880cbc-59d2-4a01-9d8c-9e2c3cd79397-kube-api-access-qjn5s\") pod \"cert-manager-cainjector-559cdd8c87-mvhgm\" (UID: \"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.214395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.214287 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" Jul 23 18:11:06.249976 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.249948 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" Jul 23 18:11:06.249976 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.249952 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frdxgl" event={"ID":"03d2e922-3d44-4e47-a482-72a6eb248b91","Type":"ContainerDied","Data":"6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3"} Jul 23 18:11:06.250175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.250006 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cbbb6687cbc553800cdaaac63d22bb89e2d315edf936e3aaa09949908d665f3" Jul 23 18:11:06.351174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:06.351139 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm"] Jul 23 18:11:06.354029 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:11:06.353996 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1880cbc_59d2_4a01_9d8c_9e2c3cd79397.slice/crio-1c8dad22cb1504eb53f73790b10eb7ad6555ae0617fd97c6b41fbb070c3e4e7a WatchSource:0}: Error finding container 1c8dad22cb1504eb53f73790b10eb7ad6555ae0617fd97c6b41fbb070c3e4e7a: Status 404 returned error can't find the container with id 1c8dad22cb1504eb53f73790b10eb7ad6555ae0617fd97c6b41fbb070c3e4e7a Jul 23 18:11:07.255555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:07.255515 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" event={"ID":"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397","Type":"ContainerStarted","Data":"1c8dad22cb1504eb53f73790b10eb7ad6555ae0617fd97c6b41fbb070c3e4e7a"} Jul 23 18:11:09.267510 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:09.267474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" event={"ID":"b1880cbc-59d2-4a01-9d8c-9e2c3cd79397","Type":"ContainerStarted","Data":"945a5ea7ee493f850a4fe28854ff42ddd6eb8e143508a8fdf4680b814b005877"} Jul 23 18:11:09.288015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:09.287958 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvhgm" podStartSLOduration=1.560013692 podStartE2EDuration="4.287940239s" totalImagesPullingTime="2.727926547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:09.287354993 +0000 UTC m=+272.854952182" watchObservedRunningTime="2026-07-23 18:11:09.287940239 +0000 UTC m=+272.855537409" Jul 23 18:11:19.207693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.207656 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh"] Jul 23 18:11:19.210040 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.210022 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.213016 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.212992 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-98wls\"" Jul 23 18:11:19.213515 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.213484 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 18:11:19.214152 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.214132 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 18:11:19.230404 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.230374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh"] Jul 23 18:11:19.371264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.371227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.371264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.371266 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.371492 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.371292 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mlg\" (UniqueName: \"kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.472337 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.472251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.472337 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.472294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.472337 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.472316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mlg\" (UniqueName: \"kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.472660 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.472632 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.472729 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.472685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.482895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.482858 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mlg\" (UniqueName: \"kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.519511 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.519469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:19.663717 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:19.663599 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh"] Jul 23 18:11:19.665939 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:11:19.665910 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc317f52a_9695_459a_843f_0b110091281d.slice/crio-f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62 WatchSource:0}: Error finding container f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62: Status 404 returned error can't find the container with id f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62 Jul 23 18:11:20.307152 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:20.307116 2570 generic.go:356] "Generic (PLEG): container finished" podID="c317f52a-9695-459a-843f-0b110091281d" containerID="61124ea2b1912779b3ac7bb418f81274623188b6e56874cc161fb13d984b1530" exitCode=0 Jul 23 18:11:20.307541 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:20.307165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" event={"ID":"c317f52a-9695-459a-843f-0b110091281d","Type":"ContainerDied","Data":"61124ea2b1912779b3ac7bb418f81274623188b6e56874cc161fb13d984b1530"} Jul 23 18:11:20.307541 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:20.307207 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" event={"ID":"c317f52a-9695-459a-843f-0b110091281d","Type":"ContainerStarted","Data":"f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62"} Jul 23 18:11:22.319117 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:22.319078 2570 generic.go:356] "Generic (PLEG): container finished" podID="c317f52a-9695-459a-843f-0b110091281d" containerID="74e5c9a62d52f43f043bbd91384289eeedf9b082cfd9b07dd179198e8f7f2449" exitCode=0 Jul 23 18:11:22.319493 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:22.319156 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" event={"ID":"c317f52a-9695-459a-843f-0b110091281d","Type":"ContainerDied","Data":"74e5c9a62d52f43f043bbd91384289eeedf9b082cfd9b07dd179198e8f7f2449"} Jul 23 18:11:23.324775 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:23.324725 2570 generic.go:356] "Generic (PLEG): container finished" podID="c317f52a-9695-459a-843f-0b110091281d" containerID="870a34eb65a2e36b49f27dcdbd7d5c4fd1eaed3795bbf42e872f8d59a8d43277" exitCode=0 Jul 23 18:11:23.325168 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:23.324813 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" event={"ID":"c317f52a-9695-459a-843f-0b110091281d","Type":"ContainerDied","Data":"870a34eb65a2e36b49f27dcdbd7d5c4fd1eaed3795bbf42e872f8d59a8d43277"} Jul 23 18:11:24.449317 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.449292 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:24.609807 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.609688 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle\") pod \"c317f52a-9695-459a-843f-0b110091281d\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " Jul 23 18:11:24.609807 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.609738 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2mlg\" (UniqueName: \"kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg\") pod \"c317f52a-9695-459a-843f-0b110091281d\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " Jul 23 18:11:24.610051 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.609819 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util\") pod \"c317f52a-9695-459a-843f-0b110091281d\" (UID: \"c317f52a-9695-459a-843f-0b110091281d\") " Jul 23 18:11:24.610495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.610456 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle" (OuterVolumeSpecName: "bundle") pod "c317f52a-9695-459a-843f-0b110091281d" (UID: "c317f52a-9695-459a-843f-0b110091281d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:11:24.612130 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.612094 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg" (OuterVolumeSpecName: "kube-api-access-j2mlg") pod "c317f52a-9695-459a-843f-0b110091281d" (UID: "c317f52a-9695-459a-843f-0b110091281d"). InnerVolumeSpecName "kube-api-access-j2mlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:11:24.615423 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.615394 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util" (OuterVolumeSpecName: "util") pod "c317f52a-9695-459a-843f-0b110091281d" (UID: "c317f52a-9695-459a-843f-0b110091281d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:11:24.710277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.710237 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:24.710277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.710270 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c317f52a-9695-459a-843f-0b110091281d-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:24.710277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:24.710281 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mlg\" (UniqueName: \"kubernetes.io/projected/c317f52a-9695-459a-843f-0b110091281d-kube-api-access-j2mlg\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:11:25.334175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:25.334144 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" Jul 23 18:11:25.334175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:25.334158 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afdjrzh" event={"ID":"c317f52a-9695-459a-843f-0b110091281d","Type":"ContainerDied","Data":"f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62"} Jul 23 18:11:25.334363 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:25.334206 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2582728ab7d4a2db489d9ac1b361a956a060ef47deebb5249d1016f0a1aeb62" Jul 23 18:11:35.395564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395526 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk"] Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395780 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="extract" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395791 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="extract" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395804 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="util" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395809 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="util" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395816 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="pull" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395821 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="pull" Jul 23 18:11:35.396023 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.395885 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c317f52a-9695-459a-843f-0b110091281d" containerName="extract" Jul 23 18:11:35.400211 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.400188 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.405020 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.404999 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 18:11:35.405152 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.405035 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 18:11:35.405964 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.405943 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 18:11:35.406080 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.406053 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 18:11:35.406190 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.406104 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bx4zl\"" Jul 23 18:11:35.406321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.406303 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 18:11:35.425683 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.425646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk"] Jul 23 18:11:35.589439 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.589399 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-metrics-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.589439 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.589445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-manager-config\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.589659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.589472 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.589659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.589532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg55\" (UniqueName: \"kubernetes.io/projected/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-kube-api-access-5pg55\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.690097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.690006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-metrics-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.690097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.690056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-manager-config\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.690097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.690081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.690306 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.690140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg55\" (UniqueName: \"kubernetes.io/projected/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-kube-api-access-5pg55\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.690704 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.690673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-manager-config\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.692577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.692548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-metrics-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.692694 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.692651 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-cert\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:35.710033 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:35.709958 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg55\" (UniqueName: \"kubernetes.io/projected/e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a-kube-api-access-5pg55\") pod \"lws-controller-manager-7676b4f7b4-l58pk\" (UID: \"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a\") " pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:36.010011 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.009915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:36.160268 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.160247 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk"] Jul 23 18:11:36.162921 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:11:36.162895 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cbc0c5_50e7_4d65_b53c_b5d6f0e6d75a.slice/crio-0e1ab803f18dadfc767042077a89c8613947b47cab70224533d5b9506eb8c98f WatchSource:0}: Error finding container 0e1ab803f18dadfc767042077a89c8613947b47cab70224533d5b9506eb8c98f: Status 404 returned error can't find the container with id 0e1ab803f18dadfc767042077a89c8613947b47cab70224533d5b9506eb8c98f Jul 23 18:11:36.381899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.381858 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" event={"ID":"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a","Type":"ContainerStarted","Data":"0e1ab803f18dadfc767042077a89c8613947b47cab70224533d5b9506eb8c98f"} Jul 23 18:11:36.544192 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.544154 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4"] Jul 23 18:11:36.546832 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.546813 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.550559 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.550534 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 18:11:36.550559 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.550553 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kmncx\"" Jul 23 18:11:36.550914 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.550894 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 18:11:36.550968 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.550943 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 18:11:36.551215 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.551199 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 18:11:36.566955 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.566924 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4"] Jul 23 18:11:36.698650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.698562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-manifests\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.698650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.698602 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-tmp\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.698650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.698636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.698889 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.698771 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmg4h\" (UniqueName: \"kubernetes.io/projected/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-kube-api-access-kmg4h\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.698889 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.698816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.799993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.799956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmg4h\" (UniqueName: \"kubernetes.io/projected/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-kube-api-access-kmg4h\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.799993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.799997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.800178 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.800030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-manifests\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.800178 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.800054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-tmp\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.800178 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.800085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.800469 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.800444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-manifests\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.802553 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.802528 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-tmp\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.802637 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.802610 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-webhook-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.802739 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.802722 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.813902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.813869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmg4h\" (UniqueName: \"kubernetes.io/projected/c0ed3ee6-a6f1-4841-b04c-df081ff26e1c-kube-api-access-kmg4h\") pod \"opendatahub-operator-controller-manager-56b5877c86-zg6g4\" (UID: \"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c\") " pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.857734 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.857690 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:36.971953 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.971859 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:11:36.980727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.980697 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:11:36.984524 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:36.984492 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 18:11:37.003984 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:37.003955 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4"] Jul 23 18:11:37.007183 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:11:37.007150 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ed3ee6_a6f1_4841_b04c_df081ff26e1c.slice/crio-fcdc42d72fdac442a4a4ba15d28dc06517b4518ac148ebe789d61a60f9ede632 WatchSource:0}: Error finding container fcdc42d72fdac442a4a4ba15d28dc06517b4518ac148ebe789d61a60f9ede632: Status 404 returned error can't find the container with id fcdc42d72fdac442a4a4ba15d28dc06517b4518ac148ebe789d61a60f9ede632 Jul 23 18:11:37.387228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:37.387186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" event={"ID":"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c","Type":"ContainerStarted","Data":"fcdc42d72fdac442a4a4ba15d28dc06517b4518ac148ebe789d61a60f9ede632"} Jul 23 18:11:40.403812 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:40.403770 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" event={"ID":"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c","Type":"ContainerStarted","Data":"cc2bbda40b2ad4c9fb92db6038e03e464710e45ab8441103866762ed13a0697e"} Jul 23 18:11:41.409619 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.409579 2570 generic.go:356] "Generic (PLEG): container finished" podID="c0ed3ee6-a6f1-4841-b04c-df081ff26e1c" containerID="cc2bbda40b2ad4c9fb92db6038e03e464710e45ab8441103866762ed13a0697e" exitCode=0 Jul 23 18:11:41.410076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.409672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" event={"ID":"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c","Type":"ContainerDied","Data":"cc2bbda40b2ad4c9fb92db6038e03e464710e45ab8441103866762ed13a0697e"} Jul 23 18:11:41.410202 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.410186 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:11:41.411288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.411265 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" event={"ID":"e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a","Type":"ContainerStarted","Data":"31622589783c803c7f0aab5d89ddbf7be561398ae12179b0867a9715af98cc8a"} Jul 23 18:11:41.411403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.411391 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:41.460255 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:41.460198 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" podStartSLOduration=1.729148208 podStartE2EDuration="6.460180077s" totalImagesPullingTime="4.731031869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:41.45859171 +0000 UTC m=+305.026188880" watchObservedRunningTime="2026-07-23 18:11:41.460180077 +0000 UTC m=+305.027777248" Jul 23 18:11:42.418461 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:42.418421 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" event={"ID":"c0ed3ee6-a6f1-4841-b04c-df081ff26e1c","Type":"ContainerStarted","Data":"d39e6fda99a475bc8a78a4611ed569e46b88ba059bcb421069bbd8b2fd907e0f"} Jul 23 18:11:42.418933 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:42.418557 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:11:42.446965 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:42.446909 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" podStartSLOduration=1.633889919 podStartE2EDuration="6.446891302s" totalImagesPullingTime="3.602616496s" totalInitContainerRuntime="1.210384887s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:11:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:11:42.445484155 +0000 UTC m=+306.013081342" watchObservedRunningTime="2026-07-23 18:11:42.446891302 +0000 UTC m=+306.014488510" Jul 23 18:11:52.420483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:52.420450 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7676b4f7b4-l58pk" Jul 23 18:11:53.424790 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:11:53.424739 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56b5877c86-zg6g4" Jul 23 18:12:10.551991 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.551954 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t"] Jul 23 18:12:10.558208 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.558176 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.561256 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.561234 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 18:12:10.561533 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.561503 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 18:12:10.561639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.561545 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-279jr\"" Jul 23 18:12:10.561639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.561566 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 18:12:10.569986 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.569936 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t"] Jul 23 18:12:10.662331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662508 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a96141-813c-41c7-bac8-54834b9fcddb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662508 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a96141-813c-41c7-bac8-54834b9fcddb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662508 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662473 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662513 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662551 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662587 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662721 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662619 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5tfz\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-kube-api-access-g5tfz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.662721 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.662688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764076 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5tfz\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-kube-api-access-g5tfz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a96141-813c-41c7-bac8-54834b9fcddb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a96141-813c-41c7-bac8-54834b9fcddb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764303 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764635 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764736 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764856 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.764955 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.764934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.765047 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.765028 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a96141-813c-41c7-bac8-54834b9fcddb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.766467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.766447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a96141-813c-41c7-bac8-54834b9fcddb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.766689 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.766673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a96141-813c-41c7-bac8-54834b9fcddb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.774688 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.774667 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5tfz\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-kube-api-access-g5tfz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.777000 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.776981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a96141-813c-41c7-bac8-54834b9fcddb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678z247t\" (UID: \"69a96141-813c-41c7-bac8-54834b9fcddb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.855565 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.855528 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4"] Jul 23 18:12:10.860576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.860552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.863250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.863224 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 18:12:10.863384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.863282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 18:12:10.868032 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.868009 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:10.875192 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.875163 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 18:12:10.875328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.875314 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s8prq\"" Jul 23 18:12:10.962549 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.962499 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4"] Jul 23 18:12:10.965527 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965527 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbng\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-kube-api-access-dtbng\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965545 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25c8216-25cc-4161-aca2-9f98a47ecec8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:10.965801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:10.965775 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.022486 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.022456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t"] Jul 23 18:12:11.024982 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:12:11.024955 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a96141_813c_41c7_bac8_54834b9fcddb.slice/crio-656e7743193452817a0e5ace60cb359be621a744ba7bc1b80103b3c244c64e12 WatchSource:0}: Error finding container 656e7743193452817a0e5ace60cb359be621a744ba7bc1b80103b3c244c64e12: Status 404 returned error can't find the container with id 656e7743193452817a0e5ace60cb359be621a744ba7bc1b80103b3c244c64e12 Jul 23 18:12:11.066902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.066860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.066909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbng\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-kube-api-access-dtbng\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.066937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.066988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.067013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25c8216-25cc-4161-aca2-9f98a47ecec8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.067062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067100 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.067083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.067863 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.067835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.069437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.069417 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c25c8216-25cc-4161-aca2-9f98a47ecec8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.069590 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.069574 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.069682 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.069663 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.069718 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.069670 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.078327 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.078304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.079478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.079455 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbng\" (UniqueName: \"kubernetes.io/projected/c25c8216-25cc-4161-aca2-9f98a47ecec8-kube-api-access-dtbng\") pod \"istiod-openshift-gateway-94bb8fbfd-94br4\" (UID: \"c25c8216-25cc-4161-aca2-9f98a47ecec8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.169712 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.169620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:11.315994 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.313592 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4"] Jul 23 18:12:11.319861 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:12:11.319835 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25c8216_25cc_4161_aca2_9f98a47ecec8.slice/crio-83f6efd595df0110e9f4489facc3f2257775546a77084189a5b50c8c3ba00cb7 WatchSource:0}: Error finding container 83f6efd595df0110e9f4489facc3f2257775546a77084189a5b50c8c3ba00cb7: Status 404 returned error can't find the container with id 83f6efd595df0110e9f4489facc3f2257775546a77084189a5b50c8c3ba00cb7 Jul 23 18:12:11.540297 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.540204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" event={"ID":"c25c8216-25cc-4161-aca2-9f98a47ecec8","Type":"ContainerStarted","Data":"83f6efd595df0110e9f4489facc3f2257775546a77084189a5b50c8c3ba00cb7"} Jul 23 18:12:11.541396 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:11.541361 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" event={"ID":"69a96141-813c-41c7-bac8-54834b9fcddb","Type":"ContainerStarted","Data":"656e7743193452817a0e5ace60cb359be621a744ba7bc1b80103b3c244c64e12"} Jul 23 18:12:15.422914 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.422874 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:12:15.423291 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.422959 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:12:15.430051 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.430023 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:12:15.430150 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.430083 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:12:15.430150 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.430109 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:12:15.565327 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.565282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" event={"ID":"c25c8216-25cc-4161-aca2-9f98a47ecec8","Type":"ContainerStarted","Data":"a6a6567bb6f7588fb0f206c2a67894365c93f4ec6c0858f30bb0c8f9dd00a792"} Jul 23 18:12:15.565514 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.565348 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:15.567124 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.567095 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" event={"ID":"69a96141-813c-41c7-bac8-54834b9fcddb","Type":"ContainerStarted","Data":"f99a38bbdb01ef2a6e8b5c11edc30411c0e0a978dd0d1ea94131618153c3ae2e"} Jul 23 18:12:15.602624 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.602568 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" podStartSLOduration=1.501924336 podStartE2EDuration="5.602551681s" totalImagesPullingTime="4.100627345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:12:15.598369786 +0000 UTC m=+339.165966955" watchObservedRunningTime="2026-07-23 18:12:15.602551681 +0000 UTC m=+339.170148830" Jul 23 18:12:15.627371 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.627308 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" podStartSLOduration=1.224465899 podStartE2EDuration="5.627288525s" totalImagesPullingTime="4.402822626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:12:15.626894839 +0000 UTC m=+339.194492037" watchObservedRunningTime="2026-07-23 18:12:15.627288525 +0000 UTC m=+339.194885701" Jul 23 18:12:15.868670 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:15.868639 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:16.574300 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:16.574210 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94br4" Jul 23 18:12:16.873802 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:16.873713 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:17.576713 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:17.576684 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:12:17.577819 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:12:17.577800 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678z247t" Jul 23 18:13:13.860678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.860644 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg"] Jul 23 18:13:13.863736 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.863718 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:13.866785 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.866754 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 18:13:13.866895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.866739 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 18:13:13.867943 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.867925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k9zht\"" Jul 23 18:13:13.875638 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.875608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg"] Jul 23 18:13:13.974667 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.974624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:13.974667 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.974673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:13.974909 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:13.974814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.075254 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.075216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.075365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.075275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.075365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.075301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.075638 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.075618 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.075707 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.075689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.087371 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.087338 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.172694 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.172595 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:14.323055 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.323028 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg"] Jul 23 18:13:14.326015 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:13:14.325987 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114801d9_2781_4dd0_b2a2_2dea008f50a6.slice/crio-c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17 WatchSource:0}: Error finding container c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17: Status 404 returned error can't find the container with id c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17 Jul 23 18:13:14.468703 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.468668 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq"] Jul 23 18:13:14.470933 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.470917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.484174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.484141 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq"] Jul 23 18:13:14.578956 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.578917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4k9n\" (UniqueName: \"kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.578956 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.578960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.579175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.579012 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.680283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.680192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.680283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.680265 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4k9n\" (UniqueName: \"kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.680449 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.680298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.680610 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.680590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.680655 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.680610 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.692455 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.692423 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4k9n\" (UniqueName: \"kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.798415 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.798374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:14.820121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.820088 2570 generic.go:356] "Generic (PLEG): container finished" podID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerID="2d78c5e3d84195275eefe2a78229e95ef85fe6a56a7a80e300d375c244eaa067" exitCode=0 Jul 23 18:13:14.820257 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.820182 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" event={"ID":"114801d9-2781-4dd0-b2a2-2dea008f50a6","Type":"ContainerDied","Data":"2d78c5e3d84195275eefe2a78229e95ef85fe6a56a7a80e300d375c244eaa067"} Jul 23 18:13:14.820257 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.820238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" event={"ID":"114801d9-2781-4dd0-b2a2-2dea008f50a6","Type":"ContainerStarted","Data":"c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17"} Jul 23 18:13:14.863904 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.863871 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb"] Jul 23 18:13:14.866564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.866540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:14.877686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.877657 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb"] Jul 23 18:13:14.940058 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.939983 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq"] Jul 23 18:13:14.942897 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:13:14.942867 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eb55f1_f0dd_4954_bdc7_86c45b2505e0.slice/crio-6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e WatchSource:0}: Error finding container 6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e: Status 404 returned error can't find the container with id 6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e Jul 23 18:13:14.983118 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.983085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:14.983118 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.983119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:14.983307 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:14.983198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wch58\" (UniqueName: \"kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.084016 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.083969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wch58\" (UniqueName: \"kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.084232 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.084038 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.084232 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.084058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.084384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.084368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.084441 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.084415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.094341 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.094306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wch58\" (UniqueName: \"kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.176675 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.176630 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:15.252593 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.252562 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs"] Jul 23 18:13:15.255156 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.255137 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.266771 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.266392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs"] Jul 23 18:13:15.285834 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.285797 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.285997 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.285867 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmgt\" (UniqueName: \"kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.285997 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.285928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.323412 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.323386 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb"] Jul 23 18:13:15.325863 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:13:15.325835 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd114a6_b51f_457c_b7a2_5ddb3782af9c.slice/crio-2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5 WatchSource:0}: Error finding container 2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5: Status 404 returned error can't find the container with id 2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5 Jul 23 18:13:15.387331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.387297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.387437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.387356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmgt\" (UniqueName: \"kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.387437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.387406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.387736 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.387708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.387866 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.387774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.397908 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.397875 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmgt\" (UniqueName: \"kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.566871 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.566801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:15.714265 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.714234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs"] Jul 23 18:13:15.734677 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:13:15.734634 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4405a7be_7b7d_46ef_b6c8_81a9b2ac9358.slice/crio-2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10 WatchSource:0}: Error finding container 2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10: Status 404 returned error can't find the container with id 2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10 Jul 23 18:13:15.827978 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.827940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerStarted","Data":"fd6515a7c54fbcd7e5e3b0ff4d71c19d0d54de2a52eaa11a4788e814067e0664"} Jul 23 18:13:15.828120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.828009 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerStarted","Data":"2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10"} Jul 23 18:13:15.829586 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.829561 2570 generic.go:356] "Generic (PLEG): container finished" podID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerID="dee78f940ae46b50a07709c96e00bd8889c67c57c8842016ad1becd1079f1971" exitCode=0 Jul 23 18:13:15.829693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.829641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" event={"ID":"50eb55f1-f0dd-4954-bdc7-86c45b2505e0","Type":"ContainerDied","Data":"dee78f940ae46b50a07709c96e00bd8889c67c57c8842016ad1becd1079f1971"} Jul 23 18:13:15.829793 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.829695 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" event={"ID":"50eb55f1-f0dd-4954-bdc7-86c45b2505e0","Type":"ContainerStarted","Data":"6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e"} Jul 23 18:13:15.831793 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.831765 2570 generic.go:356] "Generic (PLEG): container finished" podID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerID="c8936e17ed944d6feb114c536ab9f25e8e1e54b4f5489adb3090dc06f18ee666" exitCode=0 Jul 23 18:13:15.832007 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.831839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" event={"ID":"114801d9-2781-4dd0-b2a2-2dea008f50a6","Type":"ContainerDied","Data":"c8936e17ed944d6feb114c536ab9f25e8e1e54b4f5489adb3090dc06f18ee666"} Jul 23 18:13:15.833522 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.833481 2570 generic.go:356] "Generic (PLEG): container finished" podID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerID="d3abca7bb51a2ad71a7c014bf4970f3b77d7409326a3ba2f30f164926d8b7ae2" exitCode=0 Jul 23 18:13:15.833599 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.833532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" event={"ID":"2bd114a6-b51f-457c-b7a2-5ddb3782af9c","Type":"ContainerDied","Data":"d3abca7bb51a2ad71a7c014bf4970f3b77d7409326a3ba2f30f164926d8b7ae2"} Jul 23 18:13:15.833599 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:15.833582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" event={"ID":"2bd114a6-b51f-457c-b7a2-5ddb3782af9c","Type":"ContainerStarted","Data":"2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5"} Jul 23 18:13:16.840587 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.840554 2570 generic.go:356] "Generic (PLEG): container finished" podID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerID="fd6515a7c54fbcd7e5e3b0ff4d71c19d0d54de2a52eaa11a4788e814067e0664" exitCode=0 Jul 23 18:13:16.841021 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.840644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerDied","Data":"fd6515a7c54fbcd7e5e3b0ff4d71c19d0d54de2a52eaa11a4788e814067e0664"} Jul 23 18:13:16.842575 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.842549 2570 generic.go:356] "Generic (PLEG): container finished" podID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerID="b67662d9126227e885b867707dce8c0bb31aec9b03f8bd5da32c85d125d1de87" exitCode=0 Jul 23 18:13:16.842757 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.842637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" event={"ID":"50eb55f1-f0dd-4954-bdc7-86c45b2505e0","Type":"ContainerDied","Data":"b67662d9126227e885b867707dce8c0bb31aec9b03f8bd5da32c85d125d1de87"} Jul 23 18:13:16.844901 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.844877 2570 generic.go:356] "Generic (PLEG): container finished" podID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerID="ca5599e7dc1a3e5179a8ec065a9fdfc18ea6880ff8fee81ad349a9120e156291" exitCode=0 Jul 23 18:13:16.845025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.845000 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" event={"ID":"114801d9-2781-4dd0-b2a2-2dea008f50a6","Type":"ContainerDied","Data":"ca5599e7dc1a3e5179a8ec065a9fdfc18ea6880ff8fee81ad349a9120e156291"} Jul 23 18:13:16.846859 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.846838 2570 generic.go:356] "Generic (PLEG): container finished" podID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerID="a2c188b80b86f770f1fb151bfa4507b0b650e569344f60a04a7253faa7730600" exitCode=0 Jul 23 18:13:16.846936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:16.846870 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" event={"ID":"2bd114a6-b51f-457c-b7a2-5ddb3782af9c","Type":"ContainerDied","Data":"a2c188b80b86f770f1fb151bfa4507b0b650e569344f60a04a7253faa7730600"} Jul 23 18:13:17.858240 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.858199 2570 generic.go:356] "Generic (PLEG): container finished" podID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerID="74fc36e14f383446123d0860950b9fe575a696d7b75ca31ca9ab9b7821ed2d45" exitCode=0 Jul 23 18:13:17.858672 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.858283 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" event={"ID":"2bd114a6-b51f-457c-b7a2-5ddb3782af9c","Type":"ContainerDied","Data":"74fc36e14f383446123d0860950b9fe575a696d7b75ca31ca9ab9b7821ed2d45"} Jul 23 18:13:17.860035 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.860013 2570 generic.go:356] "Generic (PLEG): container finished" podID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerID="018f0a5421655de94bac6cc7185919b574af98f2de7036e089f0e02a0cb5cd40" exitCode=0 Jul 23 18:13:17.860155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.860088 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerDied","Data":"018f0a5421655de94bac6cc7185919b574af98f2de7036e089f0e02a0cb5cd40"} Jul 23 18:13:17.862176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.862154 2570 generic.go:356] "Generic (PLEG): container finished" podID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerID="e9d9688bedd47a7350f3cf595ca5e20869fc9cde743a803fd56a1b130bed5949" exitCode=0 Jul 23 18:13:17.862264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.862180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" event={"ID":"50eb55f1-f0dd-4954-bdc7-86c45b2505e0","Type":"ContainerDied","Data":"e9d9688bedd47a7350f3cf595ca5e20869fc9cde743a803fd56a1b130bed5949"} Jul 23 18:13:17.998051 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:17.998024 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:18.113653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.113564 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle\") pod \"114801d9-2781-4dd0-b2a2-2dea008f50a6\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " Jul 23 18:13:18.113653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.113608 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util\") pod \"114801d9-2781-4dd0-b2a2-2dea008f50a6\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " Jul 23 18:13:18.113653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.113635 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt\") pod \"114801d9-2781-4dd0-b2a2-2dea008f50a6\" (UID: \"114801d9-2781-4dd0-b2a2-2dea008f50a6\") " Jul 23 18:13:18.114175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.114141 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle" (OuterVolumeSpecName: "bundle") pod "114801d9-2781-4dd0-b2a2-2dea008f50a6" (UID: "114801d9-2781-4dd0-b2a2-2dea008f50a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:18.116149 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.116120 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt" (OuterVolumeSpecName: "kube-api-access-gxwxt") pod "114801d9-2781-4dd0-b2a2-2dea008f50a6" (UID: "114801d9-2781-4dd0-b2a2-2dea008f50a6"). InnerVolumeSpecName "kube-api-access-gxwxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:18.119297 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.119266 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util" (OuterVolumeSpecName: "util") pod "114801d9-2781-4dd0-b2a2-2dea008f50a6" (UID: "114801d9-2781-4dd0-b2a2-2dea008f50a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:18.215255 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.215211 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:18.215255 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.215252 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwxt\" (UniqueName: \"kubernetes.io/projected/114801d9-2781-4dd0-b2a2-2dea008f50a6-kube-api-access-gxwxt\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:18.215456 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.215266 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/114801d9-2781-4dd0-b2a2-2dea008f50a6-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:18.868561 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.868524 2570 generic.go:356] "Generic (PLEG): container finished" podID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerID="11526104137d94558dabe6f20e61be804060330f1127ff693c5d2bb20dda6b5a" exitCode=0 Jul 23 18:13:18.869020 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.868609 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerDied","Data":"11526104137d94558dabe6f20e61be804060330f1127ff693c5d2bb20dda6b5a"} Jul 23 18:13:18.870517 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.870489 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" Jul 23 18:13:18.870658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.870519 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg" event={"ID":"114801d9-2781-4dd0-b2a2-2dea008f50a6","Type":"ContainerDied","Data":"c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17"} Jul 23 18:13:18.870658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:18.870559 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c168bc70864a3bdce5f1d7c6741a2e6346647be755c8e050baf3146b2357ac17" Jul 23 18:13:19.010428 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.010404 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:19.012947 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.012927 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:19.122644 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122560 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util\") pod \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " Jul 23 18:13:19.122644 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122602 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wch58\" (UniqueName: \"kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58\") pod \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " Jul 23 18:13:19.122875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122661 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4k9n\" (UniqueName: \"kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n\") pod \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " Jul 23 18:13:19.122875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122690 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util\") pod \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " Jul 23 18:13:19.122875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122714 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle\") pod \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\" (UID: \"2bd114a6-b51f-457c-b7a2-5ddb3782af9c\") " Jul 23 18:13:19.122875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.122818 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle\") pod \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\" (UID: \"50eb55f1-f0dd-4954-bdc7-86c45b2505e0\") " Jul 23 18:13:19.123523 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.123493 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle" (OuterVolumeSpecName: "bundle") pod "50eb55f1-f0dd-4954-bdc7-86c45b2505e0" (UID: "50eb55f1-f0dd-4954-bdc7-86c45b2505e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:19.123641 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.123578 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle" (OuterVolumeSpecName: "bundle") pod "2bd114a6-b51f-457c-b7a2-5ddb3782af9c" (UID: "2bd114a6-b51f-457c-b7a2-5ddb3782af9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:19.125055 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.125024 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58" (OuterVolumeSpecName: "kube-api-access-wch58") pod "2bd114a6-b51f-457c-b7a2-5ddb3782af9c" (UID: "2bd114a6-b51f-457c-b7a2-5ddb3782af9c"). InnerVolumeSpecName "kube-api-access-wch58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:19.125148 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.125137 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n" (OuterVolumeSpecName: "kube-api-access-x4k9n") pod "50eb55f1-f0dd-4954-bdc7-86c45b2505e0" (UID: "50eb55f1-f0dd-4954-bdc7-86c45b2505e0"). InnerVolumeSpecName "kube-api-access-x4k9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:19.128650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.128626 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util" (OuterVolumeSpecName: "util") pod "50eb55f1-f0dd-4954-bdc7-86c45b2505e0" (UID: "50eb55f1-f0dd-4954-bdc7-86c45b2505e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:19.130552 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.130524 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util" (OuterVolumeSpecName: "util") pod "2bd114a6-b51f-457c-b7a2-5ddb3782af9c" (UID: "2bd114a6-b51f-457c-b7a2-5ddb3782af9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:19.224274 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224231 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wch58\" (UniqueName: \"kubernetes.io/projected/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-kube-api-access-wch58\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.224274 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224270 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4k9n\" (UniqueName: \"kubernetes.io/projected/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-kube-api-access-x4k9n\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.224274 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224285 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.224564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224298 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.224564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224309 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eb55f1-f0dd-4954-bdc7-86c45b2505e0-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.224564 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.224320 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2bd114a6-b51f-457c-b7a2-5ddb3782af9c-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:19.877311 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.877278 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" Jul 23 18:13:19.877311 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.877294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb" event={"ID":"2bd114a6-b51f-457c-b7a2-5ddb3782af9c","Type":"ContainerDied","Data":"2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5"} Jul 23 18:13:19.877820 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.877349 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f69eaaf46255f28b4821ea371a2c42330625bc305e2aa5cf089dbc99c87d5a5" Jul 23 18:13:19.879569 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.879548 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" Jul 23 18:13:19.879697 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.879595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq" event={"ID":"50eb55f1-f0dd-4954-bdc7-86c45b2505e0","Type":"ContainerDied","Data":"6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e"} Jul 23 18:13:19.879697 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:19.879628 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cb724415916e2efdb72b8e39f7987a3ff989fc7ba3f837f23a145d32f44604e" Jul 23 18:13:20.011186 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.011158 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:13:20.131095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.131006 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util\") pod \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " Jul 23 18:13:20.131095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.131049 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle\") pod \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " Jul 23 18:13:20.131276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.131125 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thmgt\" (UniqueName: \"kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt\") pod \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\" (UID: \"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358\") " Jul 23 18:13:20.131559 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.131522 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle" (OuterVolumeSpecName: "bundle") pod "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" (UID: "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:20.133300 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.133264 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt" (OuterVolumeSpecName: "kube-api-access-thmgt") pod "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" (UID: "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358"). InnerVolumeSpecName "kube-api-access-thmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:13:20.136024 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.135997 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util" (OuterVolumeSpecName: "util") pod "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" (UID: "4405a7be-7b7d-46ef-b6c8-81a9b2ac9358"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:13:20.232316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.232279 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmgt\" (UniqueName: \"kubernetes.io/projected/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-kube-api-access-thmgt\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:20.232316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.232315 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-util\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:20.232316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.232330 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4405a7be-7b7d-46ef-b6c8-81a9b2ac9358-bundle\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:13:20.886191 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.886153 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" event={"ID":"4405a7be-7b7d-46ef-b6c8-81a9b2ac9358","Type":"ContainerDied","Data":"2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10"} Jul 23 18:13:20.886544 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.886208 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca772f73e8f339c254f198a9447545606dbd003e9fad1e1995dc10b9607cb10" Jul 23 18:13:20.886544 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:13:20.886172 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs" Jul 23 18:14:28.814093 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814053 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng"] Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814317 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814327 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814338 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814343 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814354 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814359 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814365 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814370 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814377 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814382 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814389 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814393 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814401 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814407 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814412 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814417 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814422 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814426 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814432 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814438 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814445 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814450 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="util" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814454 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814459 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="pull" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814501 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eb55f1-f0dd-4954-bdc7-86c45b2505e0" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814508 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="114801d9-2781-4dd0-b2a2-2dea008f50a6" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814514 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4405a7be-7b7d-46ef-b6c8-81a9b2ac9358" containerName="extract" Jul 23 18:14:28.814536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.814519 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd114a6-b51f-457c-b7a2-5ddb3782af9c" containerName="extract" Jul 23 18:14:28.817356 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.817337 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.819925 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.819903 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pjd9f\"" Jul 23 18:14:28.830485 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.830459 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng"] Jul 23 18:14:28.903025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.902974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vf4c\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-kube-api-access-4vf4c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9e308b-9905-4033-8968-0edb50646e2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903291 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:28.903529 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:28.903367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9e308b-9905-4033-8968-0edb50646e2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004005 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.003966 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9e308b-9905-4033-8968-0edb50646e2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004005 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004029 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf4c\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-kube-api-access-4vf4c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9e308b-9905-4033-8968-0edb50646e2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004549 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004549 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004528 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004549 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.004839 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.004814 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.005141 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.005119 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a9e308b-9905-4033-8968-0edb50646e2a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.006653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.006633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a9e308b-9905-4033-8968-0edb50646e2a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.006755 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.006702 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a9e308b-9905-4033-8968-0edb50646e2a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.015545 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.015511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.015915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.015892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf4c\" (UniqueName: \"kubernetes.io/projected/6a9e308b-9905-4033-8968-0edb50646e2a-kube-api-access-4vf4c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqpng\" (UID: \"6a9e308b-9905-4033-8968-0edb50646e2a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.127105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.127071 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:29.271097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.271066 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng"] Jul 23 18:14:29.273765 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:14:29.273722 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9e308b_9905_4033_8968_0edb50646e2a.slice/crio-c519a827e3b0c1b44a56c9efbd04eee75e37fe2d8e5e0fa847ed23a4c19a9773 WatchSource:0}: Error finding container c519a827e3b0c1b44a56c9efbd04eee75e37fe2d8e5e0fa847ed23a4c19a9773: Status 404 returned error can't find the container with id c519a827e3b0c1b44a56c9efbd04eee75e37fe2d8e5e0fa847ed23a4c19a9773 Jul 23 18:14:29.275830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.275796 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:14:29.275921 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.275860 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:14:29.275921 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:29.275888 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:14:30.202560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:30.202522 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" event={"ID":"6a9e308b-9905-4033-8968-0edb50646e2a","Type":"ContainerStarted","Data":"5064ada7b6fac3897ca6b95b412731d1ac279e3c778ffb6ae0e46b2068eb0624"} Jul 23 18:14:30.202987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:30.202621 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" event={"ID":"6a9e308b-9905-4033-8968-0edb50646e2a","Type":"ContainerStarted","Data":"c519a827e3b0c1b44a56c9efbd04eee75e37fe2d8e5e0fa847ed23a4c19a9773"} Jul 23 18:14:30.231629 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:30.231566 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" podStartSLOduration=2.231549748 podStartE2EDuration="2.231549748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:14:30.229336494 +0000 UTC m=+473.796933665" watchObservedRunningTime="2026-07-23 18:14:30.231549748 +0000 UTC m=+473.799146918" Jul 23 18:14:31.127961 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:31.127923 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:31.132986 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:31.132960 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:31.207249 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:31.207216 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:14:31.208354 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:14:31.208331 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:15:58.787316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.787234 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-678f786967-bhbf4"] Jul 23 18:15:58.790328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.790305 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.792958 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.792934 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 18:15:58.793088 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.793018 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 18:15:58.794164 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.794146 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jnbq7\"" Jul 23 18:15:58.794267 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.794147 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 23 18:15:58.801862 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.801810 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-678f786967-bhbf4"] Jul 23 18:15:58.851535 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.851503 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660ffc3-0a63-4d50-b75f-6a14745c9f46-maas-api-tls\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.851704 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.851557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqtd\" (UniqueName: \"kubernetes.io/projected/2660ffc3-0a63-4d50-b75f-6a14745c9f46-kube-api-access-7kqtd\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.851704 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.851627 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660ffc3-0a63-4d50-b75f-6a14745c9f46-openshift-service-ca\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.952476 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.952436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660ffc3-0a63-4d50-b75f-6a14745c9f46-maas-api-tls\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.952666 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.952501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqtd\" (UniqueName: \"kubernetes.io/projected/2660ffc3-0a63-4d50-b75f-6a14745c9f46-kube-api-access-7kqtd\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.952666 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.952540 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660ffc3-0a63-4d50-b75f-6a14745c9f46-openshift-service-ca\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.953269 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.953248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660ffc3-0a63-4d50-b75f-6a14745c9f46-openshift-service-ca\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.955193 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.955171 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660ffc3-0a63-4d50-b75f-6a14745c9f46-maas-api-tls\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:58.964951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:58.964927 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqtd\" (UniqueName: \"kubernetes.io/projected/2660ffc3-0a63-4d50-b75f-6a14745c9f46-kube-api-access-7kqtd\") pod \"maas-api-678f786967-bhbf4\" (UID: \"2660ffc3-0a63-4d50-b75f-6a14745c9f46\") " pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:59.101171 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.101130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:15:59.255660 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.254905 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-678f786967-bhbf4"] Jul 23 18:15:59.261417 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:15:59.261380 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2660ffc3_0a63_4d50_b75f_6a14745c9f46.slice/crio-0b60cec865251dc6addde7e581f78b7ccc395a752123b7c40b3494bfa5a00a0a WatchSource:0}: Error finding container 0b60cec865251dc6addde7e581f78b7ccc395a752123b7c40b3494bfa5a00a0a: Status 404 returned error can't find the container with id 0b60cec865251dc6addde7e581f78b7ccc395a752123b7c40b3494bfa5a00a0a Jul 23 18:15:59.485722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.485627 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-c29g8"] Jul 23 18:15:59.489924 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.489895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.492464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.492435 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 18:15:59.492628 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.492553 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qrcss\"" Jul 23 18:15:59.499589 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.499561 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-c29g8"] Jul 23 18:15:59.558052 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.558005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66b6946-2d91-426e-8466-24f5075a9d8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.558244 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.558072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ckb\" (UniqueName: \"kubernetes.io/projected/d66b6946-2d91-426e-8466-24f5075a9d8d-kube-api-access-59ckb\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.610172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.610133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" event={"ID":"2660ffc3-0a63-4d50-b75f-6a14745c9f46","Type":"ContainerStarted","Data":"0b60cec865251dc6addde7e581f78b7ccc395a752123b7c40b3494bfa5a00a0a"} Jul 23 18:15:59.659402 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.659367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66b6946-2d91-426e-8466-24f5075a9d8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.659578 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.659427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ckb\" (UniqueName: \"kubernetes.io/projected/d66b6946-2d91-426e-8466-24f5075a9d8d-kube-api-access-59ckb\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.660094 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.660073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66b6946-2d91-426e-8466-24f5075a9d8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.692867 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.692839 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ckb\" (UniqueName: \"kubernetes.io/projected/d66b6946-2d91-426e-8466-24f5075a9d8d-kube-api-access-59ckb\") pod \"payload-processing-5bf47cd988-c29g8\" (UID: \"d66b6946-2d91-426e-8466-24f5075a9d8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.802251 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.802155 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:15:59.951342 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:15:59.951314 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-c29g8"] Jul 23 18:15:59.953687 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:15:59.953660 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b6946_2d91_426e_8466_24f5075a9d8d.slice/crio-5047567ccb8ea39813df8c9a417cffdcdde979930cdc830c84043dabfbdf37fb WatchSource:0}: Error finding container 5047567ccb8ea39813df8c9a417cffdcdde979930cdc830c84043dabfbdf37fb: Status 404 returned error can't find the container with id 5047567ccb8ea39813df8c9a417cffdcdde979930cdc830c84043dabfbdf37fb Jul 23 18:16:00.621111 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:00.621074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" event={"ID":"d66b6946-2d91-426e-8466-24f5075a9d8d","Type":"ContainerStarted","Data":"5047567ccb8ea39813df8c9a417cffdcdde979930cdc830c84043dabfbdf37fb"} Jul 23 18:16:03.638909 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.638869 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" event={"ID":"d66b6946-2d91-426e-8466-24f5075a9d8d","Type":"ContainerStarted","Data":"b5a04cd058f30e5b712a38a089105a4eb503e16955fbc52ca0552bcbcec97c25"} Jul 23 18:16:03.639362 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.638951 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:16:03.640507 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.640485 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" event={"ID":"2660ffc3-0a63-4d50-b75f-6a14745c9f46","Type":"ContainerStarted","Data":"b9ff3afc738153e1e1d418c655a66ff5aea7eac0fd50cd819901b89ad21221bb"} Jul 23 18:16:03.640627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.640611 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:16:03.656828 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.656735 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" podStartSLOduration=1.870169392 podStartE2EDuration="4.656715582s" totalImagesPullingTime="2.78654619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:03.656581486 +0000 UTC m=+567.224178659" watchObservedRunningTime="2026-07-23 18:16:03.656715582 +0000 UTC m=+567.224312755" Jul 23 18:16:03.677101 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:03.677021 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" podStartSLOduration=2.199997703 podStartE2EDuration="5.676995463s" totalImagesPullingTime="3.47699776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:03.674471046 +0000 UTC m=+567.242068218" watchObservedRunningTime="2026-07-23 18:16:03.676995463 +0000 UTC m=+567.244592636" Jul 23 18:16:07.670827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.670782 2570 generic.go:356] "Generic (PLEG): container finished" podID="6a9e308b-9905-4033-8968-0edb50646e2a" containerID="5064ada7b6fac3897ca6b95b412731d1ac279e3c778ffb6ae0e46b2068eb0624" exitCode=0 Jul 23 18:16:07.671242 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.670843 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" event={"ID":"6a9e308b-9905-4033-8968-0edb50646e2a","Type":"ContainerDied","Data":"5064ada7b6fac3897ca6b95b412731d1ac279e3c778ffb6ae0e46b2068eb0624"} Jul 23 18:16:07.671284 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.671261 2570 scope.go:117] "RemoveContainer" containerID="5064ada7b6fac3897ca6b95b412731d1ac279e3c778ffb6ae0e46b2068eb0624" Jul 23 18:16:07.672305 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.672279 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:16:07.672362 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.672343 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:16:07.672400 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:07.672373 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:16:08.678964 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:08.678928 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" event={"ID":"6a9e308b-9905-4033-8968-0edb50646e2a","Type":"ContainerStarted","Data":"14902133aef855ee985be3e05417917d199ef5855075ccc5638a3c840bafd6da"} Jul 23 18:16:09.128052 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:09.128007 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:16:09.128313 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:09.128288 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 23 18:16:09.128364 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:09.128341 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" podUID="6a9e308b-9905-4033-8968-0edb50646e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 23 18:16:09.658029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:09.657997 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-678f786967-bhbf4" Jul 23 18:16:10.127707 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:10.127672 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 23 18:16:10.128120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:10.127733 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" podUID="6a9e308b-9905-4033-8968-0edb50646e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 23 18:16:11.127977 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:11.127935 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 23 18:16:11.128395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:11.127994 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" podUID="6a9e308b-9905-4033-8968-0edb50646e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 23 18:16:12.128131 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:12.128093 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 23 18:16:12.128519 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:12.128155 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" podUID="6a9e308b-9905-4033-8968-0edb50646e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 23 18:16:13.131760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:13.131712 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:16:13.132145 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:13.132067 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:16:13.132817 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:13.132800 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqpng" Jul 23 18:16:14.647838 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:14.647807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-c29g8" Jul 23 18:16:37.009173 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:37.009145 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:16:37.011040 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:37.011014 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:16:46.075659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.075624 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw"] Jul 23 18:16:46.079008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.078984 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.081817 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.081798 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 18:16:46.081939 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.081844 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 18:16:46.083014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.082996 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dj6jj\"" Jul 23 18:16:46.083117 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.083025 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 23 18:16:46.090538 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.090516 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw"] Jul 23 18:16:46.172762 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.172972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.172972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.172972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.172972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvcn\" (UniqueName: \"kubernetes.io/projected/1b068fb1-045b-4e5b-920e-7d52a14ccd55-kube-api-access-9zvcn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.173129 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.172997 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274131 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274087 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274131 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274393 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274393 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274393 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvcn\" (UniqueName: \"kubernetes.io/projected/1b068fb1-045b-4e5b-920e-7d52a14ccd55-kube-api-access-9zvcn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274393 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274617 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274673 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.274763 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.274726 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.276683 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.276654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b068fb1-045b-4e5b-920e-7d52a14ccd55-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.277001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.276982 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b068fb1-045b-4e5b-920e-7d52a14ccd55-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.286896 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.286862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvcn\" (UniqueName: \"kubernetes.io/projected/1b068fb1-045b-4e5b-920e-7d52a14ccd55-kube-api-access-9zvcn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8lqw\" (UID: \"1b068fb1-045b-4e5b-920e-7d52a14ccd55\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.390054 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.389964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:46.528730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.528700 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw"] Jul 23 18:16:46.533625 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.533603 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:16:46.860996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:46.860960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" event={"ID":"1b068fb1-045b-4e5b-920e-7d52a14ccd55","Type":"ContainerStarted","Data":"87eb3688afc82b93fee2e486d077301b74ecb86cbaa10b447483bc1ccfc3f4c7"} Jul 23 18:16:47.867307 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:47.867275 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" event={"ID":"1b068fb1-045b-4e5b-920e-7d52a14ccd55","Type":"ContainerStarted","Data":"0f015b10dbc5a183ed4349bf4f894bdbbf43b9d1d3ed0d790c5812e3ff7ed3ca"} Jul 23 18:16:47.888562 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:47.888501 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" podStartSLOduration=0.735923111 podStartE2EDuration="1.888482028s" totalImagesPullingTime="1.152558917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:16:47.886450086 +0000 UTC m=+611.454047275" watchObservedRunningTime="2026-07-23 18:16:47.888482028 +0000 UTC m=+611.456079259" Jul 23 18:16:56.390111 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:56.390046 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:56.390711 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:56.390179 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:56.402639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:56.402611 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:16:56.919214 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:16:56.919175 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8lqw" Jul 23 18:20:31.017072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.017034 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:31.019344 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.019326 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.023146 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.023115 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 18:20:31.023290 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.023124 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 18:20:31.023290 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.023176 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pfz6r\"" Jul 23 18:20:31.032139 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.032112 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:31.104279 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.104234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.104279 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.104287 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.205593 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.205550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.205820 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.205610 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.207999 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.207976 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.215999 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.215974 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb\") pod \"maas-controller-7f469bfccf-zwn2j\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.329782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.329720 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:31.464526 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.464504 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:31.466521 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:20:31.466493 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df22e38_1370_4e31_a19e_8c78e193e227.slice/crio-6bb3bab4bbaea71ce76e3c779c6857903b4a34ee23d02792a8d94d868425ef95 WatchSource:0}: Error finding container 6bb3bab4bbaea71ce76e3c779c6857903b4a34ee23d02792a8d94d868425ef95: Status 404 returned error can't find the container with id 6bb3bab4bbaea71ce76e3c779c6857903b4a34ee23d02792a8d94d868425ef95 Jul 23 18:20:31.936935 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:31.936891 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" event={"ID":"9df22e38-1370-4e31-a19e-8c78e193e227","Type":"ContainerStarted","Data":"6bb3bab4bbaea71ce76e3c779c6857903b4a34ee23d02792a8d94d868425ef95"} Jul 23 18:20:33.950990 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:33.950951 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" event={"ID":"9df22e38-1370-4e31-a19e-8c78e193e227","Type":"ContainerStarted","Data":"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75"} Jul 23 18:20:33.951371 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:33.951118 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:33.970035 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:33.969981 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" podStartSLOduration=2.389755654 podStartE2EDuration="3.969959458s" totalImagesPullingTime="1.580203804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:20:33.969922538 +0000 UTC m=+837.537519710" watchObservedRunningTime="2026-07-23 18:20:33.969959458 +0000 UTC m=+837.537556631" Jul 23 18:20:44.958803 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:44.958760 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:49.593419 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:49.593372 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:49.594008 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:49.593597 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" podUID="9df22e38-1370-4e31-a19e-8c78e193e227" containerName="manager" containerID="cri-o://a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75" gracePeriod=30 Jul 23 18:20:49.937959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:49.937936 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:50.031988 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.031956 2570 generic.go:356] "Generic (PLEG): container finished" podID="9df22e38-1370-4e31-a19e-8c78e193e227" containerID="a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75" exitCode=0 Jul 23 18:20:50.032166 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.032006 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" Jul 23 18:20:50.032166 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.032048 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" event={"ID":"9df22e38-1370-4e31-a19e-8c78e193e227","Type":"ContainerDied","Data":"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75"} Jul 23 18:20:50.032166 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.032101 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-zwn2j" event={"ID":"9df22e38-1370-4e31-a19e-8c78e193e227","Type":"ContainerDied","Data":"6bb3bab4bbaea71ce76e3c779c6857903b4a34ee23d02792a8d94d868425ef95"} Jul 23 18:20:50.032166 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.032126 2570 scope.go:117] "RemoveContainer" containerID="a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75" Jul 23 18:20:50.040481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.040460 2570 scope.go:117] "RemoveContainer" containerID="a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75" Jul 23 18:20:50.040768 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:20:50.040730 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75\": container with ID starting with a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75 not found: ID does not exist" containerID="a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75" Jul 23 18:20:50.040821 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.040774 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75"} err="failed to get container status \"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75\": rpc error: code = NotFound desc = could not find container \"a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75\": container with ID starting with a3034d068311ce1b8ff1eba17303ab85c52e38231a85aad6f8f04212f8ef7d75 not found: ID does not exist" Jul 23 18:20:50.075225 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.075188 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb\") pod \"9df22e38-1370-4e31-a19e-8c78e193e227\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " Jul 23 18:20:50.075403 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.075237 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert\") pod \"9df22e38-1370-4e31-a19e-8c78e193e227\" (UID: \"9df22e38-1370-4e31-a19e-8c78e193e227\") " Jul 23 18:20:50.077506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.077470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb" (OuterVolumeSpecName: "kube-api-access-wr5wb") pod "9df22e38-1370-4e31-a19e-8c78e193e227" (UID: "9df22e38-1370-4e31-a19e-8c78e193e227"). InnerVolumeSpecName "kube-api-access-wr5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:20:50.077627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.077570 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9df22e38-1370-4e31-a19e-8c78e193e227" (UID: "9df22e38-1370-4e31-a19e-8c78e193e227"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:20:50.176175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.176084 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5wb\" (UniqueName: \"kubernetes.io/projected/9df22e38-1370-4e31-a19e-8c78e193e227-kube-api-access-wr5wb\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:20:50.176175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.176116 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9df22e38-1370-4e31-a19e-8c78e193e227-webhook-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:20:50.356074 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.356037 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:50.360468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:50.360440 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-zwn2j"] Jul 23 18:20:51.131472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.131433 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f469bfccf-8stxv"] Jul 23 18:20:51.131881 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.131758 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df22e38-1370-4e31-a19e-8c78e193e227" containerName="manager" Jul 23 18:20:51.131881 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.131773 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df22e38-1370-4e31-a19e-8c78e193e227" containerName="manager" Jul 23 18:20:51.131881 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.131844 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df22e38-1370-4e31-a19e-8c78e193e227" containerName="manager" Jul 23 18:20:51.134422 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.134403 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.136987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.136966 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 18:20:51.138410 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.138379 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pfz6r\"" Jul 23 18:20:51.138725 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.138437 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 18:20:51.143057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.143029 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df22e38-1370-4e31-a19e-8c78e193e227" path="/var/lib/kubelet/pods/9df22e38-1370-4e31-a19e-8c78e193e227/volumes" Jul 23 18:20:51.144930 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.144903 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-8stxv"] Jul 23 18:20:51.182248 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.182205 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ftd\" (UniqueName: \"kubernetes.io/projected/c8c15c67-a4f6-4d17-afd2-785ec6019412-kube-api-access-74ftd\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.182426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.182260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8c15c67-a4f6-4d17-afd2-785ec6019412-webhook-cert\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.283618 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.283582 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ftd\" (UniqueName: \"kubernetes.io/projected/c8c15c67-a4f6-4d17-afd2-785ec6019412-kube-api-access-74ftd\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.283794 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.283630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8c15c67-a4f6-4d17-afd2-785ec6019412-webhook-cert\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.286289 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.286255 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8c15c67-a4f6-4d17-afd2-785ec6019412-webhook-cert\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.294394 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.294357 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ftd\" (UniqueName: \"kubernetes.io/projected/c8c15c67-a4f6-4d17-afd2-785ec6019412-kube-api-access-74ftd\") pod \"maas-controller-7f469bfccf-8stxv\" (UID: \"c8c15c67-a4f6-4d17-afd2-785ec6019412\") " pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.445648 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.445551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:51.592712 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:51.592684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f469bfccf-8stxv"] Jul 23 18:20:51.594616 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:20:51.594580 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c15c67_a4f6_4d17_afd2_785ec6019412.slice/crio-2fdeae2ba7cda7dfceb16ef2d9f968f1aa05906a9fc20b1433dbca1a3bb64ebb WatchSource:0}: Error finding container 2fdeae2ba7cda7dfceb16ef2d9f968f1aa05906a9fc20b1433dbca1a3bb64ebb: Status 404 returned error can't find the container with id 2fdeae2ba7cda7dfceb16ef2d9f968f1aa05906a9fc20b1433dbca1a3bb64ebb Jul 23 18:20:52.044447 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:52.044408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-8stxv" event={"ID":"c8c15c67-a4f6-4d17-afd2-785ec6019412","Type":"ContainerStarted","Data":"2fdeae2ba7cda7dfceb16ef2d9f968f1aa05906a9fc20b1433dbca1a3bb64ebb"} Jul 23 18:20:53.050599 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:53.050562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f469bfccf-8stxv" event={"ID":"c8c15c67-a4f6-4d17-afd2-785ec6019412","Type":"ContainerStarted","Data":"4cc8203e70f13e3bf411cb036ca733345add294c150a132f545974d7e0116b8f"} Jul 23 18:20:53.051077 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:53.050622 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:20:53.072199 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:20:53.072132 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f469bfccf-8stxv" podStartSLOduration=1.519082884 podStartE2EDuration="2.072115485s" totalImagesPullingTime="553.032601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:20:53.070105803 +0000 UTC m=+856.637703001" watchObservedRunningTime="2026-07-23 18:20:53.072115485 +0000 UTC m=+856.639712656" Jul 23 18:21:04.058321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:21:04.058285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f469bfccf-8stxv" Jul 23 18:21:37.043546 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:21:37.043475 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:21:37.044130 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:21:37.043576 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:26:37.077477 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:26:37.077448 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:26:37.080163 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:26:37.079513 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:31:37.108408 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:31:37.108292 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:31:37.112724 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:31:37.111367 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:34:02.245952 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.245862 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:02.249178 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.249156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.251804 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.251780 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-640e6635-openshift-default-dockercfg-qkjhc\"" Jul 23 18:34:02.272302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.272266 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:02.347458 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz9v\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347466 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347585 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347611 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347639 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347965 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.347965 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.347815 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.448913 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.448874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.448913 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.448912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.448941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.448989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz9v\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449457 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449457 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449457 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.449800 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.449778 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.450114 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.450088 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.451313 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.451293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.451837 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.451820 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.460058 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.460036 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.460404 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.460386 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz9v\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v\") pod \"e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.560077 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.559990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:02.704335 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.704311 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:02.705296 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:34:02.705274 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37817ab_8415_424e_acb1_69d57a499523.slice/crio-3aa1013a6f341e69757f965e402d5d029e1a3296fa6fe6e656cd169194a7d7ce WatchSource:0}: Error finding container 3aa1013a6f341e69757f965e402d5d029e1a3296fa6fe6e656cd169194a7d7ce: Status 404 returned error can't find the container with id 3aa1013a6f341e69757f965e402d5d029e1a3296fa6fe6e656cd169194a7d7ce Jul 23 18:34:02.707440 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.707413 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:02.707520 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.707483 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:02.707520 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.707512 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:02.786657 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.786618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerStarted","Data":"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc"} Jul 23 18:34:02.786856 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.786684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerStarted","Data":"3aa1013a6f341e69757f965e402d5d029e1a3296fa6fe6e656cd169194a7d7ce"} Jul 23 18:34:02.807693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:02.807633 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podStartSLOduration=0.80760882 podStartE2EDuration="807.60882ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:34:02.805854613 +0000 UTC m=+1646.373451809" watchObservedRunningTime="2026-07-23 18:34:02.80760882 +0000 UTC m=+1646.375205991" Jul 23 18:34:03.560584 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:03.560542 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:03.565414 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:03.565390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:03.791757 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:03.791722 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:03.792782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:03.792758 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:04.235376 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.235338 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:04.238859 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.238833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.241418 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.241396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-640e6635-serving-cert\"" Jul 23 18:34:04.250264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.250234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:04.265015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.264982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.265185 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.265031 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.265185 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.265101 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.365687 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.365647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.365896 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.365701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.365896 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.365780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.365896 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:04.365816 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-serving-cert: secret "maas-api-e2e-ait-640e6635-serving-cert" not found Jul 23 18:34:04.365896 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:04.365881 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls podName:27f2829f-fba1-413e-a218-d0a4f7ba57b5 nodeName:}" failed. No retries permitted until 2026-07-23 18:34:04.865862229 +0000 UTC m=+1648.433459381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls") pod "maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" (UID: "27f2829f-fba1-413e-a218-d0a4f7ba57b5") : secret "maas-api-e2e-ait-640e6635-serving-cert" not found Jul 23 18:34:04.366463 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.366439 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.376344 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.376312 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.870645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.870599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:04.873044 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:04.873024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") pod \"maas-api-e2e-ait-640e6635-77d5dfd597-4lq48\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:05.150940 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:05.150847 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:05.292613 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:05.292590 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:05.294541 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:34:05.294504 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f2829f_fba1_413e_a218_d0a4f7ba57b5.slice/crio-629e4bb3ef9144d540e631f55ec3bf2f8e4649e7a5c40e625fae333f906a1a79 WatchSource:0}: Error finding container 629e4bb3ef9144d540e631f55ec3bf2f8e4649e7a5c40e625fae333f906a1a79: Status 404 returned error can't find the container with id 629e4bb3ef9144d540e631f55ec3bf2f8e4649e7a5c40e625fae333f906a1a79 Jul 23 18:34:05.296027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:05.296006 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:34:05.803815 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:05.803778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" event={"ID":"27f2829f-fba1-413e-a218-d0a4f7ba57b5","Type":"ContainerStarted","Data":"629e4bb3ef9144d540e631f55ec3bf2f8e4649e7a5c40e625fae333f906a1a79"} Jul 23 18:34:06.810358 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:06.810325 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" event={"ID":"27f2829f-fba1-413e-a218-d0a4f7ba57b5","Type":"ContainerStarted","Data":"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee"} Jul 23 18:34:06.810710 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:06.810436 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:06.829079 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:06.829017 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" podStartSLOduration=1.839763015 podStartE2EDuration="2.828995567s" totalImagesPullingTime="989.232552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:34:06.82846826 +0000 UTC m=+1650.396065455" watchObservedRunningTime="2026-07-23 18:34:06.828995567 +0000 UTC m=+1650.396592738" Jul 23 18:34:12.822221 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:12.822193 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:15.856004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.855969 2570 generic.go:356] "Generic (PLEG): container finished" podID="c37817ab-8415-424e-acb1-69d57a499523" containerID="e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc" exitCode=0 Jul 23 18:34:15.856421 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.856038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerDied","Data":"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc"} Jul 23 18:34:15.856464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.856423 2570 scope.go:117] "RemoveContainer" containerID="e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc" Jul 23 18:34:15.857488 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.857447 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:15.857596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.857517 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:15.857596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:15.857548 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:16.863269 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:16.863233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerStarted","Data":"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2"} Jul 23 18:34:17.560907 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:17.560865 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:17.562151 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:17.562114 2570 patch_prober.go:28] interesting pod/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 23 18:34:17.562297 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:17.562178 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 23 18:34:18.561471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:18.561431 2570 patch_prober.go:28] interesting pod/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 23 18:34:18.562089 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:18.562054 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 23 18:34:18.791947 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:18.791903 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:19.561287 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:19.561243 2570 patch_prober.go:28] interesting pod/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 23 18:34:19.561600 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:19.561572 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 23 18:34:20.561117 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:20.561074 2570 patch_prober.go:28] interesting pod/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 23 18:34:20.561439 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:20.561401 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 23 18:34:21.564625 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:21.564593 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:21.565467 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:21.565447 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:40.206882 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.206850 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:40.207486 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.207096 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" podUID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" containerName="maas-api" containerID="cri-o://49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee" gracePeriod=30 Jul 23 18:34:40.462843 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.462771 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:40.597939 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.597895 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") pod \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " Jul 23 18:34:40.598108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.597952 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk\") pod \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " Jul 23 18:34:40.598108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.597973 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca\") pod \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\" (UID: \"27f2829f-fba1-413e-a218-d0a4f7ba57b5\") " Jul 23 18:34:40.598427 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.598402 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27f2829f-fba1-413e-a218-d0a4f7ba57b5" (UID: "27f2829f-fba1-413e-a218-d0a4f7ba57b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:34:40.600220 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.600195 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27f2829f-fba1-413e-a218-d0a4f7ba57b5" (UID: "27f2829f-fba1-413e-a218-d0a4f7ba57b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:34:40.600327 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.600225 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk" (OuterVolumeSpecName: "kube-api-access-wl7gk") pod "27f2829f-fba1-413e-a218-d0a4f7ba57b5" (UID: "27f2829f-fba1-413e-a218-d0a4f7ba57b5"). InnerVolumeSpecName "kube-api-access-wl7gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:34:40.698717 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.698677 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f2829f-fba1-413e-a218-d0a4f7ba57b5-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:40.698717 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.698711 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7gk\" (UniqueName: \"kubernetes.io/projected/27f2829f-fba1-413e-a218-d0a4f7ba57b5-kube-api-access-wl7gk\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:40.698717 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.698721 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f2829f-fba1-413e-a218-d0a4f7ba57b5-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:40.989656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.989620 2570 generic.go:356] "Generic (PLEG): container finished" podID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" containerID="49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee" exitCode=0 Jul 23 18:34:40.989874 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.989662 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" Jul 23 18:34:40.989874 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.989706 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" event={"ID":"27f2829f-fba1-413e-a218-d0a4f7ba57b5","Type":"ContainerDied","Data":"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee"} Jul 23 18:34:40.989874 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.989781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48" event={"ID":"27f2829f-fba1-413e-a218-d0a4f7ba57b5","Type":"ContainerDied","Data":"629e4bb3ef9144d540e631f55ec3bf2f8e4649e7a5c40e625fae333f906a1a79"} Jul 23 18:34:40.989874 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.989813 2570 scope.go:117] "RemoveContainer" containerID="49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee" Jul 23 18:34:40.998773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.998733 2570 scope.go:117] "RemoveContainer" containerID="49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee" Jul 23 18:34:40.999082 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:40.999061 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee\": container with ID starting with 49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee not found: ID does not exist" containerID="49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee" Jul 23 18:34:40.999151 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:40.999094 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee"} err="failed to get container status \"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee\": rpc error: code = NotFound desc = could not find container \"49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee\": container with ID starting with 49681a9bb545cc86cbe0e19b0506414afc0d3081b27228d6784d1c4eb6cee9ee not found: ID does not exist" Jul 23 18:34:41.012436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:41.012401 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:41.019277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:41.019242 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-640e6635-77d5dfd597-4lq48"] Jul 23 18:34:41.142194 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:41.142162 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" path="/var/lib/kubelet/pods/27f2829f-fba1-413e-a218-d0a4f7ba57b5/volumes" Jul 23 18:34:44.960060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:44.960026 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:44.960496 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:44.960251 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" containerID="cri-o://458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2" gracePeriod=30 Jul 23 18:34:45.992498 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:45.992465 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:34:45.993024 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:45.993005 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" containerName="maas-api" Jul 23 18:34:45.993099 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:45.993028 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" containerName="maas-api" Jul 23 18:34:45.993169 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:45.993150 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f2829f-fba1-413e-a218-d0a4f7ba57b5" containerName="maas-api" Jul 23 18:34:45.997409 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:45.997389 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.000073 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.000054 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f55b464f-gw-openshift-default-dockercfg-fswxt\"" Jul 23 18:34:46.011692 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.011660 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:34:46.142295 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhzq\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142590 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.142781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.142620 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243533 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243533 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243686 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhzq\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243855 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.243911 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243868 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243906 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.243974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.244015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.244100 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244311 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.244179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.244534 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.244516 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.246272 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.246247 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.246470 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.246452 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.255209 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.255183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.255321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.255258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhzq\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq\") pod \"e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.307321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.307281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:46.447356 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.447328 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:34:46.448737 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:34:46.448680 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6eb9b5_ae2a_4dd7_808a_4453ff7174ab.slice/crio-e1ae59aa0ff8096785721502203eaac8e77f90135762280d4f815bbbada8c2ff WatchSource:0}: Error finding container e1ae59aa0ff8096785721502203eaac8e77f90135762280d4f815bbbada8c2ff: Status 404 returned error can't find the container with id e1ae59aa0ff8096785721502203eaac8e77f90135762280d4f815bbbada8c2ff Jul 23 18:34:46.450854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.450823 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:46.450946 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.450888 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:46.450946 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:46.450917 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:34:47.022064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:47.022033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerStarted","Data":"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11"} Jul 23 18:34:47.022430 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:47.022081 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerStarted","Data":"e1ae59aa0ff8096785721502203eaac8e77f90135762280d4f815bbbada8c2ff"} Jul 23 18:34:47.050172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:47.050114 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podStartSLOduration=2.05009352 podStartE2EDuration="2.05009352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:34:47.04872211 +0000 UTC m=+1690.616319314" watchObservedRunningTime="2026-07-23 18:34:47.05009352 +0000 UTC m=+1690.617690690" Jul 23 18:34:47.307897 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:47.307806 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:47.312485 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:47.312458 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:48.026483 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.026453 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:48.027447 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.027430 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:34:48.616259 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.616221 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:34:48.619796 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.619769 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.622256 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.622236 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f55b464f-serving-cert\"" Jul 23 18:34:48.633501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.633471 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:34:48.769305 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.769267 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.769509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.769317 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtcb\" (UniqueName: \"kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.769509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.769398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.870555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.870458 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.870555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.870551 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.870763 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.870583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtcb\" (UniqueName: \"kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.870763 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:48.870639 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-serving-cert: secret "maas-api-e2e-migrate-f55b464f-serving-cert" not found Jul 23 18:34:48.870763 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:48.870721 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls podName:c85271b5-2c49-4f50-848c-82159fe029c7 nodeName:}" failed. No retries permitted until 2026-07-23 18:34:49.370701551 +0000 UTC m=+1692.938298710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls") pod "maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" (UID: "c85271b5-2c49-4f50-848c-82159fe029c7") : secret "maas-api-e2e-migrate-f55b464f-serving-cert" not found Jul 23 18:34:48.871299 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.871280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:48.880434 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:48.880404 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtcb\" (UniqueName: \"kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:49.375022 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:49.374970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:49.377432 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:49.377407 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") pod \"maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:49.533925 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:49.533879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:49.681190 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:49.681163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:34:49.685173 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:34:49.685138 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85271b5_2c49_4f50_848c_82159fe029c7.slice/crio-5b9b7a81b0e0936f3669f427d349408ea54985ec4c2ae5ad50e2040c8b91e56f WatchSource:0}: Error finding container 5b9b7a81b0e0936f3669f427d349408ea54985ec4c2ae5ad50e2040c8b91e56f: Status 404 returned error can't find the container with id 5b9b7a81b0e0936f3669f427d349408ea54985ec4c2ae5ad50e2040c8b91e56f Jul 23 18:34:50.038268 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.038235 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" event={"ID":"c85271b5-2c49-4f50-848c-82159fe029c7","Type":"ContainerStarted","Data":"5b9b7a81b0e0936f3669f427d349408ea54985ec4c2ae5ad50e2040c8b91e56f"} Jul 23 18:34:50.225767 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.225676 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:50.384768 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384717 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384832 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384857 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384915 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384934 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384958 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zz9v\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.384996 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385021 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385048 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs\") pod \"c37817ab-8415-424e-acb1-69d57a499523\" (UID: \"c37817ab-8415-424e-acb1-69d57a499523\") " Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385220 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385270 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385362 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385382 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.385701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.385467 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:50.386196 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.386170 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data" (OuterVolumeSpecName: "istio-data") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:50.387280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.387251 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:34:50.387280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.387263 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token" (OuterVolumeSpecName: "istio-token") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:34:50.387280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.387274 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:34:50.387445 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.387292 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v" (OuterVolumeSpecName: "kube-api-access-5zz9v") pod "c37817ab-8415-424e-acb1-69d57a499523" (UID: "c37817ab-8415-424e-acb1-69d57a499523"). InnerVolumeSpecName "kube-api-access-5zz9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486593 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37817ab-8415-424e-acb1-69d57a499523-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486623 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37817ab-8415-424e-acb1-69d57a499523-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486633 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz9v\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-kube-api-access-5zz9v\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486643 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486655 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486663 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37817ab-8415-424e-acb1-69d57a499523-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:50.486680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:50.486670 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37817ab-8415-424e-acb1-69d57a499523-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:34:51.045280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.045243 2570 generic.go:356] "Generic (PLEG): container finished" podID="c37817ab-8415-424e-acb1-69d57a499523" containerID="458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2" exitCode=0 Jul 23 18:34:51.045495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.045300 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" Jul 23 18:34:51.045495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.045334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerDied","Data":"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2"} Jul 23 18:34:51.045495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.045399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g" event={"ID":"c37817ab-8415-424e-acb1-69d57a499523","Type":"ContainerDied","Data":"3aa1013a6f341e69757f965e402d5d029e1a3296fa6fe6e656cd169194a7d7ce"} Jul 23 18:34:51.045495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.045436 2570 scope.go:117] "RemoveContainer" containerID="458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2" Jul 23 18:34:51.047105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.047077 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" event={"ID":"c85271b5-2c49-4f50-848c-82159fe029c7","Type":"ContainerStarted","Data":"19676f5032c15f6237eca808ef005b46a8b8bd8f66a1ed0bfa76302ef51a5b28"} Jul 23 18:34:51.047252 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.047165 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:34:51.054977 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.054929 2570 scope.go:117] "RemoveContainer" containerID="e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc" Jul 23 18:34:51.063177 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.063154 2570 scope.go:117] "RemoveContainer" containerID="458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2" Jul 23 18:34:51.063432 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:51.063410 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2\": container with ID starting with 458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2 not found: ID does not exist" containerID="458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2" Jul 23 18:34:51.063482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.063436 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2"} err="failed to get container status \"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2\": rpc error: code = NotFound desc = could not find container \"458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2\": container with ID starting with 458fe239fc7226de1115b5121e79350b616d112bd1ed9f596b760d25fd216dd2 not found: ID does not exist" Jul 23 18:34:51.063482 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.063454 2570 scope.go:117] "RemoveContainer" containerID="e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc" Jul 23 18:34:51.063698 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:34:51.063683 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc\": container with ID starting with e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc not found: ID does not exist" containerID="e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc" Jul 23 18:34:51.063756 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.063703 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc"} err="failed to get container status \"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc\": rpc error: code = NotFound desc = could not find container \"e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc\": container with ID starting with e92924cece4ff01df6e7d9ba8cd1732c2f78a7722b8b4dd3a6eb9565aac568dc not found: ID does not exist" Jul 23 18:34:51.076698 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.076666 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:51.084382 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.084352 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-640e6635-openshift-default-57d9558dfc-8nb7g"] Jul 23 18:34:51.100915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.100861 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" podStartSLOduration=2.793470954 podStartE2EDuration="3.10084602s" totalImagesPullingTime="307.375066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:34:51.099871126 +0000 UTC m=+1694.667468299" watchObservedRunningTime="2026-07-23 18:34:51.10084602 +0000 UTC m=+1694.668443191" Jul 23 18:34:51.143237 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:51.143199 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c37817ab-8415-424e-acb1-69d57a499523" path="/var/lib/kubelet/pods/c37817ab-8415-424e-acb1-69d57a499523/volumes" Jul 23 18:34:57.074675 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:34:57.074633 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:35:03.028224 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.028135 2570 patch_prober.go:28] interesting pod/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 23 18:35:03.028224 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.028195 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 23 18:35:03.110309 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.110274 2570 generic.go:356] "Generic (PLEG): container finished" podID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerID="b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11" exitCode=0 Jul 23 18:35:03.110503 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.110354 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerDied","Data":"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11"} Jul 23 18:35:03.110850 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.110835 2570 scope.go:117] "RemoveContainer" containerID="b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11" Jul 23 18:35:03.111843 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.111818 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:03.111936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.111882 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:03.111936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:03.111914 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:04.117044 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:04.117012 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerStarted","Data":"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335"} Jul 23 18:35:04.307851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:04.307811 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:04.308107 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:04.308082 2570 patch_prober.go:28] interesting pod/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 23 18:35:04.308152 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:04.308135 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 23 18:35:05.307844 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:05.307804 2570 patch_prober.go:28] interesting pod/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 23 18:35:05.308201 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:05.307861 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 23 18:35:06.308250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:06.308210 2570 patch_prober.go:28] interesting pod/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 23 18:35:06.308640 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:06.308271 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 23 18:35:07.308478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:07.308437 2570 patch_prober.go:28] interesting pod/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 23 18:35:07.308915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:07.308504 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 23 18:35:08.312338 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:08.312298 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:08.312713 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:08.312658 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:08.313406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:08.313385 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:23.978825 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:23.978785 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:35:23.979336 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:23.979039 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" podUID="c85271b5-2c49-4f50-848c-82159fe029c7" containerName="maas-api" containerID="cri-o://19676f5032c15f6237eca808ef005b46a8b8bd8f66a1ed0bfa76302ef51a5b28" gracePeriod=30 Jul 23 18:35:24.215396 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.215367 2570 generic.go:356] "Generic (PLEG): container finished" podID="c85271b5-2c49-4f50-848c-82159fe029c7" containerID="19676f5032c15f6237eca808ef005b46a8b8bd8f66a1ed0bfa76302ef51a5b28" exitCode=0 Jul 23 18:35:24.215527 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.215435 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" event={"ID":"c85271b5-2c49-4f50-848c-82159fe029c7","Type":"ContainerDied","Data":"19676f5032c15f6237eca808ef005b46a8b8bd8f66a1ed0bfa76302ef51a5b28"} Jul 23 18:35:24.229715 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.229661 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:35:24.292679 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.292642 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") pod \"c85271b5-2c49-4f50-848c-82159fe029c7\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " Jul 23 18:35:24.292891 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.292699 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca\") pod \"c85271b5-2c49-4f50-848c-82159fe029c7\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " Jul 23 18:35:24.292891 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.292801 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmtcb\" (UniqueName: \"kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb\") pod \"c85271b5-2c49-4f50-848c-82159fe029c7\" (UID: \"c85271b5-2c49-4f50-848c-82159fe029c7\") " Jul 23 18:35:24.293095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.293071 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c85271b5-2c49-4f50-848c-82159fe029c7" (UID: "c85271b5-2c49-4f50-848c-82159fe029c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:35:24.294820 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.294797 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb" (OuterVolumeSpecName: "kube-api-access-jmtcb") pod "c85271b5-2c49-4f50-848c-82159fe029c7" (UID: "c85271b5-2c49-4f50-848c-82159fe029c7"). InnerVolumeSpecName "kube-api-access-jmtcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:35:24.294925 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.294815 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c85271b5-2c49-4f50-848c-82159fe029c7" (UID: "c85271b5-2c49-4f50-848c-82159fe029c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:35:24.393795 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.393729 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85271b5-2c49-4f50-848c-82159fe029c7-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:24.393795 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.393791 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85271b5-2c49-4f50-848c-82159fe029c7-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:24.394003 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:24.393811 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmtcb\" (UniqueName: \"kubernetes.io/projected/c85271b5-2c49-4f50-848c-82159fe029c7-kube-api-access-jmtcb\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:25.222176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:25.222147 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" Jul 23 18:35:25.222642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:25.222144 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992" event={"ID":"c85271b5-2c49-4f50-848c-82159fe029c7","Type":"ContainerDied","Data":"5b9b7a81b0e0936f3669f427d349408ea54985ec4c2ae5ad50e2040c8b91e56f"} Jul 23 18:35:25.222642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:25.222293 2570 scope.go:117] "RemoveContainer" containerID="19676f5032c15f6237eca808ef005b46a8b8bd8f66a1ed0bfa76302ef51a5b28" Jul 23 18:35:25.240316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:25.240286 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:35:25.244783 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:25.244756 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f55b464f-7997b9f9b5-j5992"] Jul 23 18:35:27.147019 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:27.146979 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85271b5-2c49-4f50-848c-82159fe029c7" path="/var/lib/kubelet/pods/c85271b5-2c49-4f50-848c-82159fe029c7/volumes" Jul 23 18:35:28.760368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:28.760330 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:35:28.760886 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:28.760564 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" containerID="cri-o://1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335" gracePeriod=30 Jul 23 18:35:29.578904 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.578858 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:35:29.579339 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579322 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:29.579384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579343 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:29.579384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579363 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:29.579384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579372 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:29.579481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579387 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85271b5-2c49-4f50-848c-82159fe029c7" containerName="maas-api" Jul 23 18:35:29.579481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579397 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85271b5-2c49-4f50-848c-82159fe029c7" containerName="maas-api" Jul 23 18:35:29.579481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579476 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:29.579567 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.579488 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85271b5-2c49-4f50-848c-82159fe029c7" containerName="maas-api" Jul 23 18:35:29.582171 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.582152 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.584913 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.584890 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84d59500-openshift-default-dockercfg-ss7tq\"" Jul 23 18:35:29.598509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.598465 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:35:29.739500 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739500 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdbg\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739542 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739587 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739639 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739714 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739736 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.739813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.739795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.840895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.840895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.840895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840909 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840938 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840961 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.840985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841068 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdbg\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841300 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841407 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841850 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.841850 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.841765 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.843333 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.843308 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.843444 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.843360 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.850827 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.850798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.851153 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.851132 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdbg\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg\") pod \"e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:29.891959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:29.891915 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:30.035039 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.035013 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:35:30.037046 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:35:30.037017 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb621847e_1b5d_4916_be7a_cc14744fc929.slice/crio-034a735aab08f1c48bc004753716a24952b2e749dd83bc8d47cc9942678c3319 WatchSource:0}: Error finding container 034a735aab08f1c48bc004753716a24952b2e749dd83bc8d47cc9942678c3319: Status 404 returned error can't find the container with id 034a735aab08f1c48bc004753716a24952b2e749dd83bc8d47cc9942678c3319 Jul 23 18:35:30.039152 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.039118 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:30.039266 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.039201 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:30.039266 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.039242 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:30.251658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.251618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerStarted","Data":"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013"} Jul 23 18:35:30.251881 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.251711 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerStarted","Data":"034a735aab08f1c48bc004753716a24952b2e749dd83bc8d47cc9942678c3319"} Jul 23 18:35:30.279084 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.278991 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podStartSLOduration=1.2789689819999999 podStartE2EDuration="1.278968982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:35:30.276707427 +0000 UTC m=+1733.844304599" watchObservedRunningTime="2026-07-23 18:35:30.278968982 +0000 UTC m=+1733.846566154" Jul 23 18:35:30.892894 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:30.892855 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:31.737554 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.737516 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:35:31.737909 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.737892 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c37817ab-8415-424e-acb1-69d57a499523" containerName="istio-proxy" Jul 23 18:35:31.740312 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.740289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.743111 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.743089 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-84d59500-serving-cert\"" Jul 23 18:35:31.755559 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.755529 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:35:31.861536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.861500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdnv\" (UniqueName: \"kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.861781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.861573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.861781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.861681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.897689 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.897659 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:31.962421 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.962385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.962618 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.962434 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdnv\" (UniqueName: \"kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.962711 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.962638 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.963332 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.963311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.965131 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.965114 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:31.971711 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:31.971685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdnv\" (UniqueName: \"kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv\") pod \"maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:32.051035 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:32.050940 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:32.214450 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:32.214424 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:35:32.216301 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:35:32.216273 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495c5b2b_9d56_4f19_84a9_ce9073b9a7ad.slice/crio-617f8fa57888ae7faff270f6f0362ed0c210ec6f7c6379134c6528012548bc48 WatchSource:0}: Error finding container 617f8fa57888ae7faff270f6f0362ed0c210ec6f7c6379134c6528012548bc48: Status 404 returned error can't find the container with id 617f8fa57888ae7faff270f6f0362ed0c210ec6f7c6379134c6528012548bc48 Jul 23 18:35:32.263242 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:32.263200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" event={"ID":"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad","Type":"ContainerStarted","Data":"617f8fa57888ae7faff270f6f0362ed0c210ec6f7c6379134c6528012548bc48"} Jul 23 18:35:32.263438 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:32.263421 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:32.264421 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:32.264400 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:33.269426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:33.269392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" event={"ID":"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad","Type":"ContainerStarted","Data":"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0"} Jul 23 18:35:33.269845 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:33.269492 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:33.290722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:33.290671 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" podStartSLOduration=1.904492903 podStartE2EDuration="2.290653701s" totalImagesPullingTime="386.160798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:35:33.289810969 +0000 UTC m=+1736.857408162" watchObservedRunningTime="2026-07-23 18:35:33.290653701 +0000 UTC m=+1736.858250871" Jul 23 18:35:34.016555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.016526 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:34.186855 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.186814 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.186896 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zhzq\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.186939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.186955 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.186990 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187020 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187081 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187127 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs\") pod \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\" (UID: \"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab\") " Jul 23 18:35:34.187368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187242 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:34.187525 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187290 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:35:34.187525 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187440 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:34.187525 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187456 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.187682 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187573 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:34.187950 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.187923 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:34.189262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.189234 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq" (OuterVolumeSpecName: "kube-api-access-8zhzq") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "kube-api-access-8zhzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:35:34.189437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.189416 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:35:34.189489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.189448 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:35:34.189643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.189628 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" (UID: "bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:35:34.276281 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.276248 2570 generic.go:356] "Generic (PLEG): container finished" podID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerID="1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335" exitCode=0 Jul 23 18:35:34.276678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.276300 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" Jul 23 18:35:34.276678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.276341 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerDied","Data":"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335"} Jul 23 18:35:34.276678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.276395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx" event={"ID":"bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab","Type":"ContainerDied","Data":"e1ae59aa0ff8096785721502203eaac8e77f90135762280d4f815bbbada8c2ff"} Jul 23 18:35:34.276678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.276416 2570 scope.go:117] "RemoveContainer" containerID="1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335" Jul 23 18:35:34.286198 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.286175 2570 scope.go:117] "RemoveContainer" containerID="b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11" Jul 23 18:35:34.288551 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288530 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288551 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288553 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288562 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288570 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhzq\" (UniqueName: \"kubernetes.io/projected/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-kube-api-access-8zhzq\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288590 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288598 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288607 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.288686 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.288615 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:35:34.294834 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.294817 2570 scope.go:117] "RemoveContainer" containerID="1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335" Jul 23 18:35:34.295131 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:35:34.295111 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335\": container with ID starting with 1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335 not found: ID does not exist" containerID="1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335" Jul 23 18:35:34.295177 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.295144 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335"} err="failed to get container status \"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335\": rpc error: code = NotFound desc = could not find container \"1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335\": container with ID starting with 1c530dd31655e76e93f1285d9d7e86da18223a58b02ff31625499e7fcd64f335 not found: ID does not exist" Jul 23 18:35:34.295177 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.295162 2570 scope.go:117] "RemoveContainer" containerID="b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11" Jul 23 18:35:34.295424 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:35:34.295407 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11\": container with ID starting with b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11 not found: ID does not exist" containerID="b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11" Jul 23 18:35:34.295466 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.295431 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11"} err="failed to get container status \"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11\": rpc error: code = NotFound desc = could not find container \"b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11\": container with ID starting with b74196d60ad5ac9f9f1e9b9d96c99bdd6abb46431c17ae96ad45a431be24ce11 not found: ID does not exist" Jul 23 18:35:34.300283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.300254 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:35:34.307486 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:34.307456 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f55b464f-gw-openshift-default-75dcc67995-kb9xx"] Jul 23 18:35:35.142495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:35.142457 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" path="/var/lib/kubelet/pods/bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab/volumes" Jul 23 18:35:39.283085 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:39.283055 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:35:46.341949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.341920 2570 generic.go:356] "Generic (PLEG): container finished" podID="b621847e-1b5d-4916-be7a-cc14744fc929" containerID="0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013" exitCode=0 Jul 23 18:35:46.342358 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.341991 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerDied","Data":"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013"} Jul 23 18:35:46.342402 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.342359 2570 scope.go:117] "RemoveContainer" containerID="0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013" Jul 23 18:35:46.343349 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.343325 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:46.343407 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.343394 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:46.343442 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:46.343427 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:35:47.263884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:47.263830 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:47.348987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:47.348950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerStarted","Data":"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a"} Jul 23 18:35:47.893105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:47.893068 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:47.894083 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:47.894054 2570 patch_prober.go:28] interesting pod/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:35:47.894201 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:47.894106 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 18:35:48.892499 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:48.892465 2570 patch_prober.go:28] interesting pod/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:35:48.892918 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:48.892539 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 18:35:49.892622 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:49.892584 2570 patch_prober.go:28] interesting pod/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:35:49.893025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:49.892642 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 18:35:50.893138 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:50.893101 2570 patch_prober.go:28] interesting pod/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 23 18:35:50.893538 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:50.893165 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 23 18:35:51.896985 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:51.896959 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:51.897385 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:51.897210 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:35:51.897915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:35:51.897895 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:36:07.911887 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:07.911852 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:36:07.912454 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:07.912131 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" podUID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" containerName="maas-api" containerID="cri-o://6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0" gracePeriod=30 Jul 23 18:36:08.163675 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.163614 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:36:08.290768 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.290714 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls\") pod \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " Jul 23 18:36:08.290939 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.290793 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rdnv\" (UniqueName: \"kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv\") pod \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " Jul 23 18:36:08.290939 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.290811 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca\") pod \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\" (UID: \"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad\") " Jul 23 18:36:08.291231 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.291205 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" (UID: "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:36:08.292880 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.292850 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv" (OuterVolumeSpecName: "kube-api-access-2rdnv") pod "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" (UID: "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad"). InnerVolumeSpecName "kube-api-access-2rdnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:36:08.292880 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.292871 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" (UID: "495c5b2b-9d56-4f19-84a9-ce9073b9a7ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:36:08.391437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.391401 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:08.391437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.391434 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rdnv\" (UniqueName: \"kubernetes.io/projected/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-kube-api-access-2rdnv\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:08.391437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.391446 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:08.462195 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.462108 2570 generic.go:356] "Generic (PLEG): container finished" podID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" containerID="6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0" exitCode=0 Jul 23 18:36:08.462195 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.462160 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" Jul 23 18:36:08.462195 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.462164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" event={"ID":"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad","Type":"ContainerDied","Data":"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0"} Jul 23 18:36:08.462406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.462218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz" event={"ID":"495c5b2b-9d56-4f19-84a9-ce9073b9a7ad","Type":"ContainerDied","Data":"617f8fa57888ae7faff270f6f0362ed0c210ec6f7c6379134c6528012548bc48"} Jul 23 18:36:08.462406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.462248 2570 scope.go:117] "RemoveContainer" containerID="6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0" Jul 23 18:36:08.472545 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.472525 2570 scope.go:117] "RemoveContainer" containerID="6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0" Jul 23 18:36:08.472833 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:36:08.472809 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0\": container with ID starting with 6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0 not found: ID does not exist" containerID="6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0" Jul 23 18:36:08.472910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.472841 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0"} err="failed to get container status \"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0\": rpc error: code = NotFound desc = could not find container \"6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0\": container with ID starting with 6d59e7de1767dad0c675e86ccf908b5ba2e6ff51579493f074fe57ca2f20d2f0 not found: ID does not exist" Jul 23 18:36:08.485653 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.485622 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:36:08.490228 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:08.490202 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-84d59500-86bdfd6fc7-ch2jz"] Jul 23 18:36:09.142087 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:09.142053 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" path="/var/lib/kubelet/pods/495c5b2b-9d56-4f19-84a9-ce9073b9a7ad/volumes" Jul 23 18:36:13.695509 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:13.695472 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:36:13.695956 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:13.695695 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" containerID="cri-o://21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a" gracePeriod=30 Jul 23 18:36:14.432209 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432171 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:36:14.432524 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432506 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:36:14.432596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432528 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:36:14.432596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432543 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" containerName="maas-api" Jul 23 18:36:14.432596 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432552 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" containerName="maas-api" Jul 23 18:36:14.432692 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432656 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:36:14.432692 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432671 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:36:14.432692 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.432685 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="495c5b2b-9d56-4f19-84a9-ce9073b9a7ad" containerName="maas-api" Jul 23 18:36:14.436171 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.436151 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.438400 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.438379 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-716245e6-openshift-default-dockercfg-s6466\"" Jul 23 18:36:14.452036 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.452006 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:36:14.544331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544579 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544425 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544579 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544579 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544705 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544779 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544779 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.544875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.544796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdspd\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645227 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645184 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645227 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdspd\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645630 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.645650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645639 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.646292 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645945 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.646292 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.645982 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.646292 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.646061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.646292 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.646091 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.648002 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.647976 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.648268 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.648250 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.655389 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.655365 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.655490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.655388 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdspd\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd\") pod \"e2e-derive-716245e6-openshift-default-7b95668555-6rfnb\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.746573 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.746471 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:14.888881 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.888853 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:36:14.890079 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:36:14.890045 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4158454c_1ec7_4ad4_8578_2c29f01366ce.slice/crio-7267a3c8ce8193e1f5cfb8df58b63d0859d5f609d13c095e2759dc4be4cb5f1c WatchSource:0}: Error finding container 7267a3c8ce8193e1f5cfb8df58b63d0859d5f609d13c095e2759dc4be4cb5f1c: Status 404 returned error can't find the container with id 7267a3c8ce8193e1f5cfb8df58b63d0859d5f609d13c095e2759dc4be4cb5f1c Jul 23 18:36:14.892298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.892268 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:14.892398 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.892332 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:14.892398 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:14.892361 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:15.505692 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:15.505654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerStarted","Data":"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5"} Jul 23 18:36:15.505900 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:15.505766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerStarted","Data":"7267a3c8ce8193e1f5cfb8df58b63d0859d5f609d13c095e2759dc4be4cb5f1c"} Jul 23 18:36:15.527426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:15.527370 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podStartSLOduration=1.5273509440000002 podStartE2EDuration="1.527350944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:36:15.526452055 +0000 UTC m=+1779.094049229" watchObservedRunningTime="2026-07-23 18:36:15.527350944 +0000 UTC m=+1779.094948118" Jul 23 18:36:15.747518 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:15.747480 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:15.752691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:15.752665 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:16.510895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:16.510860 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:16.511896 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:16.511875 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:18.953298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:18.953272 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:36:19.088574 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088540 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088574 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088601 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088629 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088650 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088670 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088737 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.088851 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngdbg\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.089136 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088873 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy\") pod \"b621847e-1b5d-4916-be7a-cc14744fc929\" (UID: \"b621847e-1b5d-4916-be7a-cc14744fc929\") " Jul 23 18:36:19.089136 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088908 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:36:19.089136 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.088920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:36:19.089281 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.089135 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.089281 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.089155 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.089392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.089340 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:36:19.089502 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.089391 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:36:19.089703 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.089685 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data" (OuterVolumeSpecName: "istio-data") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:36:19.091570 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.091541 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token" (OuterVolumeSpecName: "istio-token") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:36:19.091676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.091612 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:36:19.091676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.091615 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg" (OuterVolumeSpecName: "kube-api-access-ngdbg") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "kube-api-access-ngdbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:36:19.091676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.091645 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b621847e-1b5d-4916-be7a-cc14744fc929" (UID: "b621847e-1b5d-4916-be7a-cc14744fc929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:36:19.190146 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190115 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b621847e-1b5d-4916-be7a-cc14744fc929-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190146 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190144 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190146 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190152 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b621847e-1b5d-4916-be7a-cc14744fc929-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190146 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190160 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190168 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190178 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdbg\" (UniqueName: \"kubernetes.io/projected/b621847e-1b5d-4916-be7a-cc14744fc929-kube-api-access-ngdbg\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.190426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.190186 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b621847e-1b5d-4916-be7a-cc14744fc929-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:36:19.526910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.526824 2570 generic.go:356] "Generic (PLEG): container finished" podID="b621847e-1b5d-4916-be7a-cc14744fc929" containerID="21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a" exitCode=0 Jul 23 18:36:19.526910 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.526874 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" Jul 23 18:36:19.527096 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.526904 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerDied","Data":"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a"} Jul 23 18:36:19.527096 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.526960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99" event={"ID":"b621847e-1b5d-4916-be7a-cc14744fc929","Type":"ContainerDied","Data":"034a735aab08f1c48bc004753716a24952b2e749dd83bc8d47cc9942678c3319"} Jul 23 18:36:19.527096 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.526985 2570 scope.go:117] "RemoveContainer" containerID="21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a" Jul 23 18:36:19.535705 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.535681 2570 scope.go:117] "RemoveContainer" containerID="0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013" Jul 23 18:36:19.543941 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.543920 2570 scope.go:117] "RemoveContainer" containerID="21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a" Jul 23 18:36:19.544243 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:36:19.544222 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a\": container with ID starting with 21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a not found: ID does not exist" containerID="21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a" Jul 23 18:36:19.544298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.544254 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a"} err="failed to get container status \"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a\": rpc error: code = NotFound desc = could not find container \"21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a\": container with ID starting with 21072359773b868348f4a04de76ee1bfbc14b9b062af237ec6c2528505e7761a not found: ID does not exist" Jul 23 18:36:19.544298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.544273 2570 scope.go:117] "RemoveContainer" containerID="0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013" Jul 23 18:36:19.544591 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:36:19.544564 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013\": container with ID starting with 0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013 not found: ID does not exist" containerID="0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013" Jul 23 18:36:19.544728 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.544596 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013"} err="failed to get container status \"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013\": rpc error: code = NotFound desc = could not find container \"0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013\": container with ID starting with 0854b06813c1d6196eb7fdf13f6c9a2dd7b3fae9bf988a18d0963df453e86013 not found: ID does not exist" Jul 23 18:36:19.547612 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.547587 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:36:19.551995 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:19.551971 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84d59500-openshift-default-5c99494dd9-x8s99"] Jul 23 18:36:21.142592 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:21.142554 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" path="/var/lib/kubelet/pods/b621847e-1b5d-4916-be7a-cc14744fc929/volumes" Jul 23 18:36:30.594155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.594121 2570 generic.go:356] "Generic (PLEG): container finished" podID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerID="c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5" exitCode=0 Jul 23 18:36:30.594576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.594190 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerDied","Data":"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5"} Jul 23 18:36:30.594576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.594553 2570 scope.go:117] "RemoveContainer" containerID="c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5" Jul 23 18:36:30.595504 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.595482 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:30.595574 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.595548 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:30.595610 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:30.595579 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:36:31.511799 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:31.511765 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:31.601165 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:31.601128 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerStarted","Data":"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0"} Jul 23 18:36:31.746666 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:31.746632 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:31.747906 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:31.747884 2570 patch_prober.go:28] interesting pod/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 18:36:31.748013 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:31.747927 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 18:36:32.747045 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:32.747000 2570 patch_prober.go:28] interesting pod/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 18:36:32.747536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:32.747082 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 18:36:33.747829 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:33.747791 2570 patch_prober.go:28] interesting pod/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 18:36:33.748210 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:33.747858 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 18:36:34.747264 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:34.747221 2570 patch_prober.go:28] interesting pod/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 23 18:36:34.747451 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:34.747284 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 23 18:36:35.751715 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:35.751682 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:35.752172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:35.751997 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:35.752693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:35.752677 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:36:37.141407 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:37.141374 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:36:37.144461 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:37.144439 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:36:56.999319 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:56.999288 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:36:56.999726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:36:56.999499 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" containerID="cri-o://83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0" gracePeriod=30 Jul 23 18:37:02.279904 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.279876 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:37:02.386533 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386492 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386537 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386573 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386607 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386638 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386664 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.386708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386696 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.387072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386725 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdspd\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.387072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386814 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data\") pod \"4158454c-1ec7-4ad4-8578-2c29f01366ce\" (UID: \"4158454c-1ec7-4ad4-8578-2c29f01366ce\") " Jul 23 18:37:02.387072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386961 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:37:02.387072 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.386991 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:37:02.387298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.387109 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.387298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.387130 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.387298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.387192 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:37:02.387454 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.387288 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:37:02.387764 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.387718 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:37:02.388974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.388941 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:37:02.388974 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.388966 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:37:02.389368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.389331 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:37:02.389368 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.389340 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd" (OuterVolumeSpecName: "kube-api-access-vdspd") pod "4158454c-1ec7-4ad4-8578-2c29f01366ce" (UID: "4158454c-1ec7-4ad4-8578-2c29f01366ce"). InnerVolumeSpecName "kube-api-access-vdspd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:37:02.487782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487732 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.487782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487777 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.487782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487787 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdspd\" (UniqueName: \"kubernetes.io/projected/4158454c-1ec7-4ad4-8578-2c29f01366ce-kube-api-access-vdspd\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.488026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487799 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.488026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487807 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4158454c-1ec7-4ad4-8578-2c29f01366ce-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.488026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487815 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4158454c-1ec7-4ad4-8578-2c29f01366ce-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.488026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.487824 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4158454c-1ec7-4ad4-8578-2c29f01366ce-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:37:02.757659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.757611 2570 generic.go:356] "Generic (PLEG): container finished" podID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerID="83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0" exitCode=0 Jul 23 18:37:02.757893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.757674 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" Jul 23 18:37:02.757893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.757695 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerDied","Data":"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0"} Jul 23 18:37:02.757893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.757786 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb" event={"ID":"4158454c-1ec7-4ad4-8578-2c29f01366ce","Type":"ContainerDied","Data":"7267a3c8ce8193e1f5cfb8df58b63d0859d5f609d13c095e2759dc4be4cb5f1c"} Jul 23 18:37:02.757893 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.757819 2570 scope.go:117] "RemoveContainer" containerID="83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0" Jul 23 18:37:02.767830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.767807 2570 scope.go:117] "RemoveContainer" containerID="c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5" Jul 23 18:37:02.776702 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.776677 2570 scope.go:117] "RemoveContainer" containerID="83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0" Jul 23 18:37:02.777076 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:37:02.777056 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0\": container with ID starting with 83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0 not found: ID does not exist" containerID="83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0" Jul 23 18:37:02.777124 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.777087 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0"} err="failed to get container status \"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0\": rpc error: code = NotFound desc = could not find container \"83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0\": container with ID starting with 83b0ddbe5015b4c50810b5437d6c74a27a6c45bb69d2d55de3c1406b4be669e0 not found: ID does not exist" Jul 23 18:37:02.777124 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.777107 2570 scope.go:117] "RemoveContainer" containerID="c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5" Jul 23 18:37:02.777372 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:37:02.777350 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5\": container with ID starting with c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5 not found: ID does not exist" containerID="c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5" Jul 23 18:37:02.777425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.777379 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5"} err="failed to get container status \"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5\": rpc error: code = NotFound desc = could not find container \"c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5\": container with ID starting with c73b5ac6371c9468026c8a72735486bf3de5b4c14ce84a19c028fc794f3645e5 not found: ID does not exist" Jul 23 18:37:02.781758 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.781715 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:37:02.786185 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:02.786149 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-716245e6-openshift-default-7b95668555-6rfnb"] Jul 23 18:37:03.142436 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:37:03.142400 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" path="/var/lib/kubelet/pods/4158454c-1ec7-4ad4-8578-2c29f01366ce/volumes" Jul 23 18:38:40.532811 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.532779 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533077 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533088 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6eb9b5-ae2a-4dd7-808a-4453ff7174ab" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533101 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533107 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533112 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533120 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533134 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533139 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533183 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533191 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533197 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.533331 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.533205 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b621847e-1b5d-4916-be7a-cc14744fc929" containerName="istio-proxy" Jul 23 18:38:40.536001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.535985 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.539515 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.539490 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b6f3e049-openshift-default-dockercfg-nkxpt\"" Jul 23 18:38:40.570178 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.570146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:38:40.593691 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593782 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593800 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhkj\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593820 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.593873 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.594079 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.594079 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.593954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694387 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694446 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694475 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694502 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhkj\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694673 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.694915 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.695025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.694967 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.695025 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.695003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.695143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.695065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.695143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.695079 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.695484 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.695461 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.697082 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.697060 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.697290 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.697273 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.705435 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.705410 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.705640 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.705617 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhkj\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj\") pod \"e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.846461 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.846427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:40.995899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.995875 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:38:40.997162 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:38:40.997133 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20ce234_258b_477d_a220_6b99d9112396.slice/crio-a4dca778112927d6f8b300e076c64932ccd1093628d1a9e56ba27872d7f08153 WatchSource:0}: Error finding container a4dca778112927d6f8b300e076c64932ccd1093628d1a9e56ba27872d7f08153: Status 404 returned error can't find the container with id a4dca778112927d6f8b300e076c64932ccd1093628d1a9e56ba27872d7f08153 Jul 23 18:38:40.999057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.999030 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:38:40.999122 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.999105 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:38:40.999162 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:40.999135 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:38:41.234936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:41.234844 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerStarted","Data":"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65"} Jul 23 18:38:41.234936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:41.234914 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerStarted","Data":"a4dca778112927d6f8b300e076c64932ccd1093628d1a9e56ba27872d7f08153"} Jul 23 18:38:41.258617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:41.258560 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podStartSLOduration=1.2585422849999999 podStartE2EDuration="1.258542285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:38:41.256045092 +0000 UTC m=+1924.823642295" watchObservedRunningTime="2026-07-23 18:38:41.258542285 +0000 UTC m=+1924.826139455" Jul 23 18:38:41.846569 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:41.846526 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:42.851936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:42.851905 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:43.244525 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.244435 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:43.245567 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.245546 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:38:43.513836 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.513736 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:38:43.514194 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.514176 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:43.514237 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.514197 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4158454c-1ec7-4ad4-8578-2c29f01366ce" containerName="istio-proxy" Jul 23 18:38:43.516708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.516686 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.519081 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.519055 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b6f3e049-serving-cert\"" Jul 23 18:38:43.530696 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.530667 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:38:43.618560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.618521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sdg\" (UniqueName: \"kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.618737 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.618585 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.618737 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.618645 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.720114 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.720079 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sdg\" (UniqueName: \"kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.720293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.720155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.720293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.720180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.720293 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:38:43.720285 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-serving-cert: secret "maas-api-e2e-mt-b6f3e049-serving-cert" not found Jul 23 18:38:43.720404 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:38:43.720335 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls podName:82a0006b-97b3-4f23-a7c3-41b948f868b5 nodeName:}" failed. No retries permitted until 2026-07-23 18:38:44.220318922 +0000 UTC m=+1927.787916072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls") pod "maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" (UID: "82a0006b-97b3-4f23-a7c3-41b948f868b5") : secret "maas-api-e2e-mt-b6f3e049-serving-cert" not found Jul 23 18:38:43.720837 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.720816 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:43.732775 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:43.732716 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sdg\" (UniqueName: \"kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:44.224727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:44.224685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:44.228247 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:44.228217 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") pod \"maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:44.427919 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:44.427873 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:44.575611 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:44.575583 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:38:44.577379 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:38:44.577348 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a0006b_97b3_4f23_a7c3_41b948f868b5.slice/crio-315d1e0906f28c61aae374d1e739a0e710d9fb866b8a9fd53a6a15c561de4b53 WatchSource:0}: Error finding container 315d1e0906f28c61aae374d1e739a0e710d9fb866b8a9fd53a6a15c561de4b53: Status 404 returned error can't find the container with id 315d1e0906f28c61aae374d1e739a0e710d9fb866b8a9fd53a6a15c561de4b53 Jul 23 18:38:45.257027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:45.256929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" event={"ID":"82a0006b-97b3-4f23-a7c3-41b948f868b5","Type":"ContainerStarted","Data":"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20"} Jul 23 18:38:45.257406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:45.257155 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:45.257406 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:45.257201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" event={"ID":"82a0006b-97b3-4f23-a7c3-41b948f868b5","Type":"ContainerStarted","Data":"315d1e0906f28c61aae374d1e739a0e710d9fb866b8a9fd53a6a15c561de4b53"} Jul 23 18:38:45.276028 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:45.275975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" podStartSLOduration=1.897867529 podStartE2EDuration="2.275955408s" totalImagesPullingTime="378.087879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:38:45.273769926 +0000 UTC m=+1928.841367092" watchObservedRunningTime="2026-07-23 18:38:45.275955408 +0000 UTC m=+1928.843552579" Jul 23 18:38:51.267635 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:51.267599 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:38:58.624302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:58.623714 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:43878->10.134.0.40:15021: read: connection reset by peer" start-of-body= Jul 23 18:38:58.624302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:58.623804 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:43878->10.134.0.40:15021: read: connection reset by peer" Jul 23 18:38:59.332887 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.332852 2570 generic.go:356] "Generic (PLEG): container finished" podID="d20ce234-258b-477d-a220-6b99d9112396" containerID="6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65" exitCode=0 Jul 23 18:38:59.333063 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.332895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerDied","Data":"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65"} Jul 23 18:38:59.333261 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.333246 2570 scope.go:117] "RemoveContainer" containerID="6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65" Jul 23 18:38:59.334257 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.334224 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:38:59.334353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.334309 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:38:59.334415 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:38:59.334355 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:00.339695 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:00.339658 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerStarted","Data":"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f"} Jul 23 18:39:00.846963 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:00.846924 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:00.847165 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:00.847142 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:39:00.847219 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:00.847193 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 18:39:01.847701 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:01.847657 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:39:01.848143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:01.847725 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 18:39:02.847710 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:02.847664 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:39:02.848139 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:02.847727 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 18:39:03.847646 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:03.847600 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 23 18:39:03.847898 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:03.847663 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 23 18:39:04.850612 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:04.850580 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:04.851112 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:04.850995 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:04.851689 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:04.851672 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:05.816941 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:05.816902 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:39:05.817141 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:05.817115 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" podUID="82a0006b-97b3-4f23-a7c3-41b948f868b5" containerName="maas-api" containerID="cri-o://aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20" gracePeriod=30 Jul 23 18:39:06.068842 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.068777 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:39:06.229104 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.229064 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4sdg\" (UniqueName: \"kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg\") pod \"82a0006b-97b3-4f23-a7c3-41b948f868b5\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " Jul 23 18:39:06.229302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.229144 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca\") pod \"82a0006b-97b3-4f23-a7c3-41b948f868b5\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " Jul 23 18:39:06.229302 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.229179 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") pod \"82a0006b-97b3-4f23-a7c3-41b948f868b5\" (UID: \"82a0006b-97b3-4f23-a7c3-41b948f868b5\") " Jul 23 18:39:06.229700 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.229494 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82a0006b-97b3-4f23-a7c3-41b948f868b5" (UID: "82a0006b-97b3-4f23-a7c3-41b948f868b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:39:06.229700 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.229572 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82a0006b-97b3-4f23-a7c3-41b948f868b5-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:06.231300 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.231280 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg" (OuterVolumeSpecName: "kube-api-access-n4sdg") pod "82a0006b-97b3-4f23-a7c3-41b948f868b5" (UID: "82a0006b-97b3-4f23-a7c3-41b948f868b5"). InnerVolumeSpecName "kube-api-access-n4sdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:39:06.231383 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.231367 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82a0006b-97b3-4f23-a7c3-41b948f868b5" (UID: "82a0006b-97b3-4f23-a7c3-41b948f868b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:39:06.330813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.330781 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4sdg\" (UniqueName: \"kubernetes.io/projected/82a0006b-97b3-4f23-a7c3-41b948f868b5-kube-api-access-n4sdg\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:06.330813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.330814 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82a0006b-97b3-4f23-a7c3-41b948f868b5-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:06.376097 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.376063 2570 generic.go:356] "Generic (PLEG): container finished" podID="82a0006b-97b3-4f23-a7c3-41b948f868b5" containerID="aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20" exitCode=0 Jul 23 18:39:06.376283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.376108 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" Jul 23 18:39:06.376283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.376137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" event={"ID":"82a0006b-97b3-4f23-a7c3-41b948f868b5","Type":"ContainerDied","Data":"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20"} Jul 23 18:39:06.376283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.376179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr" event={"ID":"82a0006b-97b3-4f23-a7c3-41b948f868b5","Type":"ContainerDied","Data":"315d1e0906f28c61aae374d1e739a0e710d9fb866b8a9fd53a6a15c561de4b53"} Jul 23 18:39:06.376283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.376208 2570 scope.go:117] "RemoveContainer" containerID="aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20" Jul 23 18:39:06.385068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.385049 2570 scope.go:117] "RemoveContainer" containerID="aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20" Jul 23 18:39:06.385373 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:39:06.385352 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20\": container with ID starting with aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20 not found: ID does not exist" containerID="aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20" Jul 23 18:39:06.385453 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.385385 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20"} err="failed to get container status \"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20\": rpc error: code = NotFound desc = could not find container \"aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20\": container with ID starting with aca893a96cab80504a8b3fc03a742137704124dec5a3e9191aedc2f6ae9bfd20 not found: ID does not exist" Jul 23 18:39:06.398814 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.398788 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:39:06.405361 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:06.404552 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b6f3e049-59b49744f4-p45kr"] Jul 23 18:39:07.142839 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:07.142803 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a0006b-97b3-4f23-a7c3-41b948f868b5" path="/var/lib/kubelet/pods/82a0006b-97b3-4f23-a7c3-41b948f868b5/volumes" Jul 23 18:39:18.110055 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:18.110009 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:39:18.110583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:18.110307 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" containerID="cri-o://c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f" gracePeriod=30 Jul 23 18:39:19.853951 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:19.853915 2570 patch_prober.go:28] interesting pod/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:39:19.854341 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:19.853974 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:39:23.368087 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.368061 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:23.460789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.460689 2570 generic.go:356] "Generic (PLEG): container finished" podID="d20ce234-258b-477d-a220-6b99d9112396" containerID="c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f" exitCode=0 Jul 23 18:39:23.460789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.460733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerDied","Data":"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f"} Jul 23 18:39:23.460789 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.460737 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" Jul 23 18:39:23.461026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.460792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7" event={"ID":"d20ce234-258b-477d-a220-6b99d9112396","Type":"ContainerDied","Data":"a4dca778112927d6f8b300e076c64932ccd1093628d1a9e56ba27872d7f08153"} Jul 23 18:39:23.461026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.460814 2570 scope.go:117] "RemoveContainer" containerID="c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f" Jul 23 18:39:23.469489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.469471 2570 scope.go:117] "RemoveContainer" containerID="6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65" Jul 23 18:39:23.471561 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471540 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471589 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471622 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471648 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471680 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471725 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471784 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rhkj\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.471830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471812 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.472036 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471855 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy\") pod \"d20ce234-258b-477d-a220-6b99d9112396\" (UID: \"d20ce234-258b-477d-a220-6b99d9112396\") " Jul 23 18:39:23.472036 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.471906 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:23.472160 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.472105 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.472160 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.472107 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:23.472160 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.472116 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:23.472367 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.472347 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data" (OuterVolumeSpecName: "istio-data") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:23.472437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.472380 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:39:23.474174 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.474151 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:39:23.474280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.474153 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:39:23.474280 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.474238 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token" (OuterVolumeSpecName: "istio-token") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:39:23.474646 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.474623 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj" (OuterVolumeSpecName: "kube-api-access-4rhkj") pod "d20ce234-258b-477d-a220-6b99d9112396" (UID: "d20ce234-258b-477d-a220-6b99d9112396"). InnerVolumeSpecName "kube-api-access-4rhkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:39:23.478011 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.477990 2570 scope.go:117] "RemoveContainer" containerID="c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f" Jul 23 18:39:23.478319 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:39:23.478292 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f\": container with ID starting with c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f not found: ID does not exist" containerID="c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f" Jul 23 18:39:23.478373 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.478321 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f"} err="failed to get container status \"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f\": rpc error: code = NotFound desc = could not find container \"c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f\": container with ID starting with c74b8b270d6699b748fd0d6fca74470c7f4a2b11f0273fb754327a0d87b0ec0f not found: ID does not exist" Jul 23 18:39:23.478373 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.478340 2570 scope.go:117] "RemoveContainer" containerID="6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65" Jul 23 18:39:23.478586 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:39:23.478571 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65\": container with ID starting with 6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65 not found: ID does not exist" containerID="6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65" Jul 23 18:39:23.478626 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.478590 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65"} err="failed to get container status \"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65\": rpc error: code = NotFound desc = could not find container \"6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65\": container with ID starting with 6639a6dea9109cc47484819b98644772ecdaf8c85250604623ef451fbc652b65 not found: ID does not exist" Jul 23 18:39:23.573250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573209 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573239 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573249 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573250 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573258 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573531 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573267 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d20ce234-258b-477d-a220-6b99d9112396-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573531 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573276 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d20ce234-258b-477d-a220-6b99d9112396-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573531 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573284 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rhkj\" (UniqueName: \"kubernetes.io/projected/d20ce234-258b-477d-a220-6b99d9112396-kube-api-access-4rhkj\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.573531 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.573324 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d20ce234-258b-477d-a220-6b99d9112396-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:39:23.783027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.782993 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:39:23.787641 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:23.787611 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b6f3e049-openshift-default-96dff4d99-wqcd7"] Jul 23 18:39:25.142211 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:25.142166 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20ce234-258b-477d-a220-6b99d9112396" path="/var/lib/kubelet/pods/d20ce234-258b-477d-a220-6b99d9112396/volumes" Jul 23 18:39:38.923343 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923298 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923618 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923631 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923640 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a0006b-97b3-4f23-a7c3-41b948f868b5" containerName="maas-api" Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923646 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a0006b-97b3-4f23-a7c3-41b948f868b5" containerName="maas-api" Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923719 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:38.923726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.923730 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a0006b-97b3-4f23-a7c3-41b948f868b5" containerName="maas-api" Jul 23 18:39:38.928138 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.928122 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:38.930610 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.930589 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-368d7465-openshift-default-dockercfg-j25z9\"" Jul 23 18:39:38.937781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:38.937735 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:39:39.013389 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.013330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68wp\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.013718 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.013685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.013928 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.013910 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014182 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014143 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014293 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.014488 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.014303 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.114912 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.114871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68wp\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.114943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.114969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.114996 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115013 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115530 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115396 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115611 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115663 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115612 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.115663 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.115637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.116222 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.116195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.117642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.117614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.117800 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.117784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.124657 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.124637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.125453 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.125431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68wp\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp\") pod \"e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.239227 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.239130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:39.390027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.389999 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:39:39.392236 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:39:39.392205 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3334c2fb_ffb7_48f2_872c_f19f843a449d.slice/crio-acd683a03090544c7a558dcb8630e09965f9e0bf519b6bb6dca430cb2646d396 WatchSource:0}: Error finding container acd683a03090544c7a558dcb8630e09965f9e0bf519b6bb6dca430cb2646d396: Status 404 returned error can't find the container with id acd683a03090544c7a558dcb8630e09965f9e0bf519b6bb6dca430cb2646d396 Jul 23 18:39:39.394347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.394314 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:39.394431 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.394378 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:39.394431 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.394405 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:39.545862 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.545758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerStarted","Data":"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496"} Jul 23 18:39:39.545862 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.545830 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerStarted","Data":"acd683a03090544c7a558dcb8630e09965f9e0bf519b6bb6dca430cb2646d396"} Jul 23 18:39:39.568270 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:39.568209 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podStartSLOduration=1.568190947 podStartE2EDuration="1.568190947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:39.567033659 +0000 UTC m=+1983.134630830" watchObservedRunningTime="2026-07-23 18:39:39.568190947 +0000 UTC m=+1983.135788117" Jul 23 18:39:40.239937 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:40.239900 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:40.245161 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:40.245131 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:40.551133 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:40.551041 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:40.552215 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:40.552193 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:41.724599 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.724565 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:39:41.724978 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.724923 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:41.724978 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.724939 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:41.725060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.725000 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20ce234-258b-477d-a220-6b99d9112396" containerName="istio-proxy" Jul 23 18:39:41.727992 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.727974 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.730783 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.730756 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-368d7465-serving-cert\"" Jul 23 18:39:41.740424 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.740398 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:39:41.838656 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.838618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flg5p\" (UniqueName: \"kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.838864 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.838708 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.838864 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.838780 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.939266 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.939232 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.939411 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.939306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flg5p\" (UniqueName: \"kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.939411 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.939343 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.939966 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.939934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.941732 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.941715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:41.949170 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:41.949146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flg5p\" (UniqueName: \"kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p\") pod \"maas-api-e2e-mt-368d7465-6896b66cf6-m64wv\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:42.039242 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:42.039148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:42.182216 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:42.182186 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:39:42.183858 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:39:42.183832 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30554504_8d17_41e8_8ea0_f796e6f1b211.slice/crio-3063d203a9b711bc339e15835c53ab983b080cfa033bc6814b39731ff198e203 WatchSource:0}: Error finding container 3063d203a9b711bc339e15835c53ab983b080cfa033bc6814b39731ff198e203: Status 404 returned error can't find the container with id 3063d203a9b711bc339e15835c53ab983b080cfa033bc6814b39731ff198e203 Jul 23 18:39:42.185425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:42.185406 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:39:42.564015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:42.563971 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" event={"ID":"30554504-8d17-41e8-8ea0-f796e6f1b211","Type":"ContainerStarted","Data":"3063d203a9b711bc339e15835c53ab983b080cfa033bc6814b39731ff198e203"} Jul 23 18:39:43.571059 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:43.571018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" event={"ID":"30554504-8d17-41e8-8ea0-f796e6f1b211","Type":"ContainerStarted","Data":"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721"} Jul 23 18:39:43.571468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:43.571158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:43.590590 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:43.590533 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" podStartSLOduration=2.120987672 podStartE2EDuration="2.590516606s" totalImagesPullingTime="469.528934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:39:43.589923271 +0000 UTC m=+1987.157520479" watchObservedRunningTime="2026-07-23 18:39:43.590516606 +0000 UTC m=+1987.158113777" Jul 23 18:39:49.581628 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:49.581595 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:39:54.625242 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.625211 2570 generic.go:356] "Generic (PLEG): container finished" podID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerID="e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496" exitCode=0 Jul 23 18:39:54.625672 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.625266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerDied","Data":"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496"} Jul 23 18:39:54.625672 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.625643 2570 scope.go:117] "RemoveContainer" containerID="e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496" Jul 23 18:39:54.626760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.626722 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:54.626835 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.626800 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:54.626835 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:54.626831 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:39:55.551455 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:55.551399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:55.632552 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:55.632506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerStarted","Data":"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a"} Jul 23 18:39:56.240137 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:56.240094 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:39:56.241443 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:56.241417 2570 patch_prober.go:28] interesting pod/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 18:39:56.241552 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:56.241482 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 18:39:57.240060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:57.240016 2570 patch_prober.go:28] interesting pod/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 18:39:57.240451 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:57.240102 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 18:39:58.240125 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:58.240083 2570 patch_prober.go:28] interesting pod/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 18:39:58.240524 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:58.240143 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 18:39:59.240455 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:59.240414 2570 patch_prober.go:28] interesting pod/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 23 18:39:59.240899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:39:59.240485 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 23 18:40:00.243598 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:00.243567 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:40:00.244064 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:00.243860 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:40:00.244632 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:00.244612 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:40:40.138392 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.138356 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:40:40.138916 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.138593 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" podUID="30554504-8d17-41e8-8ea0-f796e6f1b211" containerName="maas-api" containerID="cri-o://6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721" gracePeriod=30 Jul 23 18:40:40.392034 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.391962 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:40:40.482979 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.482938 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls\") pod \"30554504-8d17-41e8-8ea0-f796e6f1b211\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " Jul 23 18:40:40.483186 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.483081 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flg5p\" (UniqueName: \"kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p\") pod \"30554504-8d17-41e8-8ea0-f796e6f1b211\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " Jul 23 18:40:40.483186 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.483144 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca\") pod \"30554504-8d17-41e8-8ea0-f796e6f1b211\" (UID: \"30554504-8d17-41e8-8ea0-f796e6f1b211\") " Jul 23 18:40:40.483535 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.483502 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30554504-8d17-41e8-8ea0-f796e6f1b211" (UID: "30554504-8d17-41e8-8ea0-f796e6f1b211"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:40:40.485093 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.485067 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p" (OuterVolumeSpecName: "kube-api-access-flg5p") pod "30554504-8d17-41e8-8ea0-f796e6f1b211" (UID: "30554504-8d17-41e8-8ea0-f796e6f1b211"). InnerVolumeSpecName "kube-api-access-flg5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:40:40.485194 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.485090 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30554504-8d17-41e8-8ea0-f796e6f1b211" (UID: "30554504-8d17-41e8-8ea0-f796e6f1b211"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:40:40.584643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.584607 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flg5p\" (UniqueName: \"kubernetes.io/projected/30554504-8d17-41e8-8ea0-f796e6f1b211-kube-api-access-flg5p\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:40:40.584643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.584638 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30554504-8d17-41e8-8ea0-f796e6f1b211-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:40:40.584643 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.584648 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30554504-8d17-41e8-8ea0-f796e6f1b211-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:40:40.856096 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.856063 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" Jul 23 18:40:40.856304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.856100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" event={"ID":"30554504-8d17-41e8-8ea0-f796e6f1b211","Type":"ContainerDied","Data":"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721"} Jul 23 18:40:40.856304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.856160 2570 scope.go:117] "RemoveContainer" containerID="6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721" Jul 23 18:40:40.856304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.856098 2570 generic.go:356] "Generic (PLEG): container finished" podID="30554504-8d17-41e8-8ea0-f796e6f1b211" containerID="6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721" exitCode=0 Jul 23 18:40:40.856304 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.856208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv" event={"ID":"30554504-8d17-41e8-8ea0-f796e6f1b211","Type":"ContainerDied","Data":"3063d203a9b711bc339e15835c53ab983b080cfa033bc6814b39731ff198e203"} Jul 23 18:40:40.865521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.865496 2570 scope.go:117] "RemoveContainer" containerID="6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721" Jul 23 18:40:40.865900 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:40:40.865877 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721\": container with ID starting with 6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721 not found: ID does not exist" containerID="6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721" Jul 23 18:40:40.865971 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.865910 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721"} err="failed to get container status \"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721\": rpc error: code = NotFound desc = could not find container \"6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721\": container with ID starting with 6e0a169150030d68d342e6e68d9db61fef9c4971a6df0e35516542892c6ef721 not found: ID does not exist" Jul 23 18:40:40.879671 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.879642 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:40:40.886434 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:40.886404 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-368d7465-6896b66cf6-m64wv"] Jul 23 18:40:41.142835 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:40:41.142735 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30554504-8d17-41e8-8ea0-f796e6f1b211" path="/var/lib/kubelet/pods/30554504-8d17-41e8-8ea0-f796e6f1b211/volumes" Jul 23 18:41:37.155636 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:37.155605 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:41:37.156240 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:37.155841 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" containerID="cri-o://c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a" gracePeriod=30 Jul 23 18:41:37.174001 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:37.173968 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:41:37.178014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:37.177991 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:41:42.400415 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.400388 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:41:42.415899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415824 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.415899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415865 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.415899 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415891 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r68wp\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415905 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415924 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.415951 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416016 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416049 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416155 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416096 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert\") pod \"3334c2fb-ffb7-48f2-872c-f19f843a449d\" (UID: \"3334c2fb-ffb7-48f2-872c-f19f843a449d\") " Jul 23 18:41:42.416478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416198 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:42.416478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416347 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.416478 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416373 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:42.416649 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416523 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:42.416709 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416687 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:41:42.416781 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.416756 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:42.418256 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.418224 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp" (OuterVolumeSpecName: "kube-api-access-r68wp") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "kube-api-access-r68wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:41:42.418383 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.418252 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:41:42.418383 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.418259 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:41:42.418383 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.418352 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3334c2fb-ffb7-48f2-872c-f19f843a449d" (UID: "3334c2fb-ffb7-48f2-872c-f19f843a449d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:41:42.516845 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516814 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.516845 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516845 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68wp\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-kube-api-access-r68wp\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.516845 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516855 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.517076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516863 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.517076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516872 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.517076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516881 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3334c2fb-ffb7-48f2-872c-f19f843a449d-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.517076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516888 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3334c2fb-ffb7-48f2-872c-f19f843a449d-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:42.517076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:42.516896 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3334c2fb-ffb7-48f2-872c-f19f843a449d-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:41:43.168565 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.168523 2570 generic.go:356] "Generic (PLEG): container finished" podID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerID="c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a" exitCode=0 Jul 23 18:41:43.168727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.168590 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" Jul 23 18:41:43.168727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.168601 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerDied","Data":"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a"} Jul 23 18:41:43.168727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.168681 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm" event={"ID":"3334c2fb-ffb7-48f2-872c-f19f843a449d","Type":"ContainerDied","Data":"acd683a03090544c7a558dcb8630e09965f9e0bf519b6bb6dca430cb2646d396"} Jul 23 18:41:43.168727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.168703 2570 scope.go:117] "RemoveContainer" containerID="c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a" Jul 23 18:41:43.177277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.177254 2570 scope.go:117] "RemoveContainer" containerID="e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496" Jul 23 18:41:43.186138 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.185855 2570 scope.go:117] "RemoveContainer" containerID="c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a" Jul 23 18:41:43.189029 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:41:43.188998 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a\": container with ID starting with c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a not found: ID does not exist" containerID="c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a" Jul 23 18:41:43.189164 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.189039 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a"} err="failed to get container status \"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a\": rpc error: code = NotFound desc = could not find container \"c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a\": container with ID starting with c6b95e6f1dcb8d7c24b871c078d64a06350e18853a7a4eda801238aae0ee652a not found: ID does not exist" Jul 23 18:41:43.189164 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.189064 2570 scope.go:117] "RemoveContainer" containerID="e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496" Jul 23 18:41:43.189548 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:41:43.189521 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496\": container with ID starting with e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496 not found: ID does not exist" containerID="e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496" Jul 23 18:41:43.189602 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.189555 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496"} err="failed to get container status \"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496\": rpc error: code = NotFound desc = could not find container \"e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496\": container with ID starting with e131f8df88cbda289a10ce4be455123d7e0eb31115e7818fb8f68ddb626c9496 not found: ID does not exist" Jul 23 18:41:43.222901 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.222862 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:41:43.229993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:43.229960 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-368d7465-openshift-default-86bf8f7b67-lcbpm"] Jul 23 18:41:45.142277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:41:45.142245 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" path="/var/lib/kubelet/pods/3334c2fb-ffb7-48f2-872c-f19f843a449d/volumes" Jul 23 18:42:38.489774 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.489719 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490060 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30554504-8d17-41e8-8ea0-f796e6f1b211" containerName="maas-api" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490072 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30554504-8d17-41e8-8ea0-f796e6f1b211" containerName="maas-api" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490086 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490091 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490102 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490108 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490156 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="30554504-8d17-41e8-8ea0-f796e6f1b211" containerName="maas-api" Jul 23 18:42:38.490288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.490164 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:38.494170 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.494146 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.496867 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.496838 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-47f36d-openshift-default-dockercfg-9pfjh\"" Jul 23 18:42:38.506780 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.506731 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:42:38.601057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601285 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601285 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601284 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601321 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktztq\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601407 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.601471 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.601431 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702140 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702140 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702159 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktztq\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702425 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702565 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702678 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.702936 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702923 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.703037 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.702961 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.704844 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.704817 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.705024 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.705003 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.714776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.714470 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.714776 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.714559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktztq\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq\") pod \"e2e-inf-a-47f36d-openshift-default-5776994b77-mk795\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.804528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.804410 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:38.957359 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.957330 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:42:38.959196 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:42:38.959164 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9ebb6b_c4b1_4ef7_bc5e_d261f565018e.slice/crio-0da068e550c420e455f4336a7bb84fab28e74cd64d3879985556a3214fcaec2e WatchSource:0}: Error finding container 0da068e550c420e455f4336a7bb84fab28e74cd64d3879985556a3214fcaec2e: Status 404 returned error can't find the container with id 0da068e550c420e455f4336a7bb84fab28e74cd64d3879985556a3214fcaec2e Jul 23 18:42:38.961445 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.961409 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:38.961511 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.961481 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:38.961548 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:38.961509 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:39.436506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:39.436465 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerStarted","Data":"2ed193e2ce1dcf661e244785b38c0bbdb2704256039c76be28b197e649da4824"} Jul 23 18:42:39.436702 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:39.436521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerStarted","Data":"0da068e550c420e455f4336a7bb84fab28e74cd64d3879985556a3214fcaec2e"} Jul 23 18:42:39.466011 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:39.465957 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podStartSLOduration=1.465937898 podStartE2EDuration="1.465937898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:42:39.46317635 +0000 UTC m=+2163.030773521" watchObservedRunningTime="2026-07-23 18:42:39.465937898 +0000 UTC m=+2163.033535069" Jul 23 18:42:39.805035 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:39.804941 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:40.810207 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:40.810175 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:41.446153 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.446117 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:41.447137 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.447120 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:41.546635 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.546597 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:42:41.547225 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.547201 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3334c2fb-ffb7-48f2-872c-f19f843a449d" containerName="istio-proxy" Jul 23 18:42:41.550186 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.550164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.552791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.552765 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-47f36d-serving-cert\"" Jul 23 18:42:41.566120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.566086 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:42:41.627119 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.627074 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.627353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.627133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.627353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.627203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwwd\" (UniqueName: \"kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.728187 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.728090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.728187 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.728144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwwd\" (UniqueName: \"kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.728398 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.728235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.729084 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.729053 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.730720 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.730693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.740607 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.740578 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwwd\" (UniqueName: \"kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd\") pod \"maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:41.861664 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:41.861620 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:42.002568 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:42.002542 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:42:42.005068 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:42:42.005032 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9590252e_54b8_4fba_a388_08ea7e7388ed.slice/crio-144d679f2511ee87db699178441d86fa1ff5b395bab8f0abf94338ec57a4ee60 WatchSource:0}: Error finding container 144d679f2511ee87db699178441d86fa1ff5b395bab8f0abf94338ec57a4ee60: Status 404 returned error can't find the container with id 144d679f2511ee87db699178441d86fa1ff5b395bab8f0abf94338ec57a4ee60 Jul 23 18:42:42.452663 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:42.452557 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" event={"ID":"9590252e-54b8-4fba-a388-08ea7e7388ed","Type":"ContainerStarted","Data":"144d679f2511ee87db699178441d86fa1ff5b395bab8f0abf94338ec57a4ee60"} Jul 23 18:42:43.459225 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:43.459187 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" event={"ID":"9590252e-54b8-4fba-a388-08ea7e7388ed","Type":"ContainerStarted","Data":"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4"} Jul 23 18:42:43.464672 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:43.460365 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:43.486213 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:43.486156 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" podStartSLOduration=2.037818353 podStartE2EDuration="2.486137235s" totalImagesPullingTime="448.318882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:42:43.484274957 +0000 UTC m=+2167.051872176" watchObservedRunningTime="2026-07-23 18:42:43.486137235 +0000 UTC m=+2167.053734405" Jul 23 18:42:50.477625 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:50.477595 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:42:52.835244 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.835201 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:42:52.839222 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.839194 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:52.841948 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.841920 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d06db7-serving-cert\"" Jul 23 18:42:52.850528 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.850455 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:42:52.932849 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.932811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:52.933076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.932866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8fd\" (UniqueName: \"kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:52.933076 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:52.932981 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.033931 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.033877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8fd\" (UniqueName: \"kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.034143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.034006 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.034143 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.034091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.034799 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.034738 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.036584 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.036552 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.044848 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.044818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8fd\" (UniqueName: \"kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd\") pod \"maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.152196 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.152113 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:53.302344 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.302305 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:42:53.304016 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:42:53.303987 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea1a462_4edc_48d0_9eb4_30dac76dcbf7.slice/crio-57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2 WatchSource:0}: Error finding container 57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2: Status 404 returned error can't find the container with id 57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2 Jul 23 18:42:53.512554 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:53.512459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" event={"ID":"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7","Type":"ContainerStarted","Data":"57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2"} Jul 23 18:42:54.519606 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:54.519567 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" event={"ID":"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7","Type":"ContainerStarted","Data":"ecf8d59527318368505c95219789ab6de2f61c1dbda2456c7a46763b5943f5eb"} Jul 23 18:42:54.520022 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:54.519632 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:42:54.540054 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:54.539998 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" podStartSLOduration=2.201884395 podStartE2EDuration="2.539978362s" totalImagesPullingTime="338.093967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:42:54.537847335 +0000 UTC m=+2178.105444507" watchObservedRunningTime="2026-07-23 18:42:54.539978362 +0000 UTC m=+2178.107575534" Jul 23 18:42:55.527455 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.527416 2570 generic.go:356] "Generic (PLEG): container finished" podID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerID="2ed193e2ce1dcf661e244785b38c0bbdb2704256039c76be28b197e649da4824" exitCode=0 Jul 23 18:42:55.527966 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.527488 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerDied","Data":"2ed193e2ce1dcf661e244785b38c0bbdb2704256039c76be28b197e649da4824"} Jul 23 18:42:55.527966 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.527952 2570 scope.go:117] "RemoveContainer" containerID="2ed193e2ce1dcf661e244785b38c0bbdb2704256039c76be28b197e649da4824" Jul 23 18:42:55.528892 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.528862 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:55.529004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.528929 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:55.529004 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:55.528964 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:42:56.446657 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:56.446614 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:56.534614 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:56.534580 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerStarted","Data":"ccba37a06ee461b913c390f6c3368e772b2a821ed2343c5380f2015631d75fb7"} Jul 23 18:42:56.804625 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:56.804497 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:42:56.805993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:56.805964 2570 patch_prober.go:28] interesting pod/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:42:56.806120 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:56.806028 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 18:42:57.805053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:57.805005 2570 patch_prober.go:28] interesting pod/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:42:57.805464 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:57.805070 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 18:42:58.805673 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:58.805635 2570 patch_prober.go:28] interesting pod/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:42:58.806099 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:58.805689 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 18:42:59.805387 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:59.805338 2570 patch_prober.go:28] interesting pod/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 23 18:42:59.805609 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:42:59.805409 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 23 18:43:00.533654 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:43:00.533626 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:43:00.809083 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:43:00.809000 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:43:00.809370 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:43:00.809349 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:43:00.810009 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:43:00.809987 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:44:29.045286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.045241 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:44:29.045921 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.045580 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" podUID="9590252e-54b8-4fba-a388-08ea7e7388ed" containerName="maas-api" containerID="cri-o://c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4" gracePeriod=30 Jul 23 18:44:29.298614 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.298554 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:44:29.407098 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.407059 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls\") pod \"9590252e-54b8-4fba-a388-08ea7e7388ed\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " Jul 23 18:44:29.407271 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.407113 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdwwd\" (UniqueName: \"kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd\") pod \"9590252e-54b8-4fba-a388-08ea7e7388ed\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " Jul 23 18:44:29.407271 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.407166 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca\") pod \"9590252e-54b8-4fba-a388-08ea7e7388ed\" (UID: \"9590252e-54b8-4fba-a388-08ea7e7388ed\") " Jul 23 18:44:29.407611 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.407587 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9590252e-54b8-4fba-a388-08ea7e7388ed" (UID: "9590252e-54b8-4fba-a388-08ea7e7388ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:44:29.409290 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.409258 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9590252e-54b8-4fba-a388-08ea7e7388ed" (UID: "9590252e-54b8-4fba-a388-08ea7e7388ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:44:29.409353 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.409258 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd" (OuterVolumeSpecName: "kube-api-access-bdwwd") pod "9590252e-54b8-4fba-a388-08ea7e7388ed" (UID: "9590252e-54b8-4fba-a388-08ea7e7388ed"). InnerVolumeSpecName "kube-api-access-bdwwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:44:29.508972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.508932 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9590252e-54b8-4fba-a388-08ea7e7388ed-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:29.508972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.508968 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwwd\" (UniqueName: \"kubernetes.io/projected/9590252e-54b8-4fba-a388-08ea7e7388ed-kube-api-access-bdwwd\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:29.508972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:29.508979 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9590252e-54b8-4fba-a388-08ea7e7388ed-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:30.007369 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.007334 2570 generic.go:356] "Generic (PLEG): container finished" podID="9590252e-54b8-4fba-a388-08ea7e7388ed" containerID="c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4" exitCode=0 Jul 23 18:44:30.007581 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.007378 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" Jul 23 18:44:30.007581 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.007422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" event={"ID":"9590252e-54b8-4fba-a388-08ea7e7388ed","Type":"ContainerDied","Data":"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4"} Jul 23 18:44:30.007581 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.007481 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp" event={"ID":"9590252e-54b8-4fba-a388-08ea7e7388ed","Type":"ContainerDied","Data":"144d679f2511ee87db699178441d86fa1ff5b395bab8f0abf94338ec57a4ee60"} Jul 23 18:44:30.007581 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.007510 2570 scope.go:117] "RemoveContainer" containerID="c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4" Jul 23 18:44:30.016159 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.016141 2570 scope.go:117] "RemoveContainer" containerID="c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4" Jul 23 18:44:30.016417 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:44:30.016399 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4\": container with ID starting with c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4 not found: ID does not exist" containerID="c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4" Jul 23 18:44:30.016465 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.016425 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4"} err="failed to get container status \"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4\": rpc error: code = NotFound desc = could not find container \"c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4\": container with ID starting with c73a2285664496662c927f51ea95115b4316e3a1aa3a4bb75cf526b1bf7eb6a4 not found: ID does not exist" Jul 23 18:44:30.031014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.030987 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:44:30.035793 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:30.035769 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-47f36d-674c8bf7bf-2kzkp"] Jul 23 18:44:31.143523 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:31.143485 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9590252e-54b8-4fba-a388-08ea7e7388ed" path="/var/lib/kubelet/pods/9590252e-54b8-4fba-a388-08ea7e7388ed/volumes" Jul 23 18:44:40.874871 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:40.874839 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:44:40.875308 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:40.875070 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" containerID="cri-o://ccba37a06ee461b913c390f6c3368e772b2a821ed2343c5380f2015631d75fb7" gracePeriod=30 Jul 23 18:44:45.812075 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:45.812029 2570 patch_prober.go:28] interesting pod/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:44:45.812453 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:45.812100 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:44:46.090863 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.090830 2570 generic.go:356] "Generic (PLEG): container finished" podID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerID="ccba37a06ee461b913c390f6c3368e772b2a821ed2343c5380f2015631d75fb7" exitCode=0 Jul 23 18:44:46.091048 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.090900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerDied","Data":"ccba37a06ee461b913c390f6c3368e772b2a821ed2343c5380f2015631d75fb7"} Jul 23 18:44:46.091048 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.090962 2570 scope.go:117] "RemoveContainer" containerID="2ed193e2ce1dcf661e244785b38c0bbdb2704256039c76be28b197e649da4824" Jul 23 18:44:46.127714 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.127686 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:44:46.263547 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263510 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.263730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263587 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.263730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263603 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.263730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263631 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.263730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263656 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.263730 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.264029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263835 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktztq\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.264029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263873 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.264029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263899 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo\") pod \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\" (UID: \"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e\") " Jul 23 18:44:46.264029 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.263924 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:44:46.264239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264077 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:44:46.264239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264160 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:44:46.264239 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264199 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:44:46.264388 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264330 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.264515 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264494 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.264645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264570 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.264645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.264620 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:44:46.266082 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.266053 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:44:46.266190 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.266135 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:44:46.266318 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.266298 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:44:46.266577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.266552 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq" (OuterVolumeSpecName: "kube-api-access-ktztq") pod "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" (UID: "fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e"). InnerVolumeSpecName "kube-api-access-ktztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365448 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktztq\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-kube-api-access-ktztq\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365481 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365492 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365500 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365510 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:46.365540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:46.365518 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:44:47.097379 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.097323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" event={"ID":"fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e","Type":"ContainerDied","Data":"0da068e550c420e455f4336a7bb84fab28e74cd64d3879985556a3214fcaec2e"} Jul 23 18:44:47.097829 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.097435 2570 scope.go:117] "RemoveContainer" containerID="ccba37a06ee461b913c390f6c3368e772b2a821ed2343c5380f2015631d75fb7" Jul 23 18:44:47.097829 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.097352 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795" Jul 23 18:44:47.128126 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.128099 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:44:47.132202 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.132175 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-47f36d-openshift-default-5776994b77-mk795"] Jul 23 18:44:47.142068 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:44:47.142037 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" path="/var/lib/kubelet/pods/fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e/volumes" Jul 23 18:45:01.926659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:01.926621 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:45:01.927151 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:01.926883 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" podUID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" containerName="maas-api" containerID="cri-o://ecf8d59527318368505c95219789ab6de2f61c1dbda2456c7a46763b5943f5eb" gracePeriod=30 Jul 23 18:45:02.169515 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.169486 2570 generic.go:356] "Generic (PLEG): container finished" podID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" containerID="ecf8d59527318368505c95219789ab6de2f61c1dbda2456c7a46763b5943f5eb" exitCode=0 Jul 23 18:45:02.169664 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.169559 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" event={"ID":"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7","Type":"ContainerDied","Data":"ecf8d59527318368505c95219789ab6de2f61c1dbda2456c7a46763b5943f5eb"} Jul 23 18:45:02.169664 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.169623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" event={"ID":"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7","Type":"ContainerDied","Data":"57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2"} Jul 23 18:45:02.169664 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.169646 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57248e9677f90f8d534e9cbe8497658596e4cd9f0580257f516017855ffa1be2" Jul 23 18:45:02.173098 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.173080 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:45:02.204613 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.204535 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca\") pod \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " Jul 23 18:45:02.204613 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.204569 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls\") pod \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " Jul 23 18:45:02.204846 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.204693 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g8fd\" (UniqueName: \"kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd\") pod \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\" (UID: \"3ea1a462-4edc-48d0-9eb4-30dac76dcbf7\") " Jul 23 18:45:02.204962 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.204943 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" (UID: "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:45:02.206854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.206824 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd" (OuterVolumeSpecName: "kube-api-access-5g8fd") pod "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" (UID: "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7"). InnerVolumeSpecName "kube-api-access-5g8fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:45:02.206963 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.206906 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" (UID: "3ea1a462-4edc-48d0-9eb4-30dac76dcbf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:45:02.305501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.305455 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8fd\" (UniqueName: \"kubernetes.io/projected/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-kube-api-access-5g8fd\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:02.305501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.305495 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:02.305501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:02.305506 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:03.174472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:03.174442 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx" Jul 23 18:45:03.203270 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:03.203238 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:45:03.207229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:03.207198 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d06db7-8ffd4fc55-jmshx"] Jul 23 18:45:05.142707 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:05.142674 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" path="/var/lib/kubelet/pods/3ea1a462-4edc-48d0-9eb4-30dac76dcbf7/volumes" Jul 23 18:45:14.980895 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.980858 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981299 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981336 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981348 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" containerName="maas-api" Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981356 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" containerName="maas-api" Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981367 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:14.981381 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981375 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:14.981680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981385 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9590252e-54b8-4fba-a388-08ea7e7388ed" containerName="maas-api" Jul 23 18:45:14.981680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981394 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9590252e-54b8-4fba-a388-08ea7e7388ed" containerName="maas-api" Jul 23 18:45:14.981680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981488 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:14.981680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981501 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9590252e-54b8-4fba-a388-08ea7e7388ed" containerName="maas-api" Jul 23 18:45:14.981680 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.981512 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea1a462-4edc-48d0-9eb4-30dac76dcbf7" containerName="maas-api" Jul 23 18:45:14.984283 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.984261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:14.987182 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.987162 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dc11fe-openshift-default-dockercfg-nb6zl\"" Jul 23 18:45:14.998079 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:14.998054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:45:15.119469 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119549 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119787 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.119902 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.119827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlqg\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.220922 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.220882 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.220929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.220952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.220973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.220989 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221039 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221121 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlqg\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221151 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221426 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221646 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221587 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221711 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.221972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.221952 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.223404 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.223384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.223627 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.223609 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.231787 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.231706 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.231787 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.231732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlqg\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg\") pod \"e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.294165 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.294123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:15.437177 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.437149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:45:15.438956 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:45:15.438926 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4b3b2f_75d8_4001_8e99_82cce84fb90c.slice/crio-72a27323db7bb36b6d4681c0a10dcae999417d34c28c0e80dbc1615172d160d7 WatchSource:0}: Error finding container 72a27323db7bb36b6d4681c0a10dcae999417d34c28c0e80dbc1615172d160d7: Status 404 returned error can't find the container with id 72a27323db7bb36b6d4681c0a10dcae999417d34c28c0e80dbc1615172d160d7 Jul 23 18:45:15.440976 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.440941 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:15.441028 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.441013 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:15.441073 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:15.441041 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:16.239314 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:16.239273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerStarted","Data":"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e"} Jul 23 18:45:16.239689 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:16.239337 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerStarted","Data":"72a27323db7bb36b6d4681c0a10dcae999417d34c28c0e80dbc1615172d160d7"} Jul 23 18:45:16.267332 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:16.267281 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podStartSLOduration=2.267261816 podStartE2EDuration="2.267261816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:16.265727447 +0000 UTC m=+2319.833324617" watchObservedRunningTime="2026-07-23 18:45:16.267261816 +0000 UTC m=+2319.834858985" Jul 23 18:45:16.294281 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:16.294237 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:16.299723 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:16.299693 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:17.244799 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:17.244770 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:17.245859 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:17.245840 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:18.554512 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.554481 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:18.554916 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.554902 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9ebb6b-c4b1-4ef7-bc5e-d261f565018e" containerName="istio-proxy" Jul 23 18:45:18.557837 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.557815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.562430 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.562404 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-dc11fe-serving-cert\"" Jul 23 18:45:18.575534 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.575499 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:18.650346 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.650308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.650346 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.650346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmvv\" (UniqueName: \"kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.650573 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.650409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.751323 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.751289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.751323 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.751327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmvv\" (UniqueName: \"kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.751589 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.751365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.751589 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:45:18.751473 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-serving-cert: secret "maas-api-e2e-api-a-dc11fe-serving-cert" not found Jul 23 18:45:18.751589 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:45:18.751527 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls podName:9e429235-e0cf-420e-b944-9470d2354681 nodeName:}" failed. No retries permitted until 2026-07-23 18:45:19.251511084 +0000 UTC m=+2322.819108233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls") pod "maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" (UID: "9e429235-e0cf-420e-b944-9470d2354681") : secret "maas-api-e2e-api-a-dc11fe-serving-cert" not found Jul 23 18:45:18.752086 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.752060 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:18.768576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:18.768546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmvv\" (UniqueName: \"kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:19.255279 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:19.255250 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:19.257647 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:19.257616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") pod \"maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:19.468394 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:19.468362 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:19.620637 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:45:19.620606 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e429235_e0cf_420e_b944_9470d2354681.slice/crio-f4f0a886a2ab709ecaf3a522709b4f8f9e63c5fa282999cd72f51313e2b9c189 WatchSource:0}: Error finding container f4f0a886a2ab709ecaf3a522709b4f8f9e63c5fa282999cd72f51313e2b9c189: Status 404 returned error can't find the container with id f4f0a886a2ab709ecaf3a522709b4f8f9e63c5fa282999cd72f51313e2b9c189 Jul 23 18:45:19.620982 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:19.620784 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:19.622138 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:19.622118 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 18:45:20.262822 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:20.262781 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" event={"ID":"9e429235-e0cf-420e-b944-9470d2354681","Type":"ContainerStarted","Data":"0727c37a98d7bbe3b9d3735200bc26ad75396f1bd9deea6952fcad118dee84fa"} Jul 23 18:45:20.263015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:20.262836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" event={"ID":"9e429235-e0cf-420e-b944-9470d2354681","Type":"ContainerStarted","Data":"f4f0a886a2ab709ecaf3a522709b4f8f9e63c5fa282999cd72f51313e2b9c189"} Jul 23 18:45:20.263015 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:20.262865 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:20.287169 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:20.287114 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" podStartSLOduration=1.758422022 podStartE2EDuration="2.287097993s" totalImagesPullingTime="528.675971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:20.285063854 +0000 UTC m=+2323.852661023" watchObservedRunningTime="2026-07-23 18:45:20.287097993 +0000 UTC m=+2323.854695222" Jul 23 18:45:26.276875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.276842 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:26.683929 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.683890 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:45:26.688009 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.687978 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:45:26.688179 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.688084 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.690669 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.690651 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-81e4dc-serving-cert\"" Jul 23 18:45:26.826576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.826539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.826576 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.826583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk47f\" (UniqueName: \"kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.826838 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.826688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.927113 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.927064 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.927113 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.927116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk47f\" (UniqueName: \"kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.927342 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.927197 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.927342 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:45:26.927233 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-serving-cert: secret "maas-api-e2e-api-b-81e4dc-serving-cert" not found Jul 23 18:45:26.927342 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:45:26.927323 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls podName:b3d78fbb-374c-47a5-9ee9-a9023adaa079 nodeName:}" failed. No retries permitted until 2026-07-23 18:45:27.427300738 +0000 UTC m=+2330.994897887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls") pod "maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" (UID: "b3d78fbb-374c-47a5-9ee9-a9023adaa079") : secret "maas-api-e2e-api-b-81e4dc-serving-cert" not found Jul 23 18:45:26.927801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.927784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:26.939299 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:26.939216 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk47f\" (UniqueName: \"kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:27.431968 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:27.431924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:27.434571 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:27.434546 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") pod \"maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:27.598765 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:27.598699 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:27.786577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:27.786533 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:45:28.303468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:28.303387 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" event={"ID":"b3d78fbb-374c-47a5-9ee9-a9023adaa079","Type":"ContainerStarted","Data":"5055687321b88e1b0eed2e56e62dcf054410a15c08053ece5edc3227fa73fc30"} Jul 23 18:45:29.310014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:29.309972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" event={"ID":"b3d78fbb-374c-47a5-9ee9-a9023adaa079","Type":"ContainerStarted","Data":"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f"} Jul 23 18:45:29.310395 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:29.310097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:29.338017 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:29.337962 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" podStartSLOduration=2.737589803 podStartE2EDuration="3.337944487s" totalImagesPullingTime="600.354684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:45:29.337698455 +0000 UTC m=+2332.905295627" watchObservedRunningTime="2026-07-23 18:45:29.337944487 +0000 UTC m=+2332.905541658" Jul 23 18:45:32.246828 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.246784 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 18:45:32.247287 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.246864 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 18:45:32.326462 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.326430 2570 generic.go:356] "Generic (PLEG): container finished" podID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerID="e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e" exitCode=0 Jul 23 18:45:32.326645 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.326514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerDied","Data":"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e"} Jul 23 18:45:32.326934 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.326920 2570 scope.go:117] "RemoveContainer" containerID="e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e" Jul 23 18:45:32.327892 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.327867 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:32.327961 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.327947 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:32.328002 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:32.327993 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:45:33.333629 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:33.333598 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerStarted","Data":"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6"} Jul 23 18:45:34.294609 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:34.294567 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:34.294849 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:34.294828 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 18:45:34.294901 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:34.294875 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 18:45:35.295126 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:35.295087 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 18:45:35.295534 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:35.295144 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 18:45:35.321722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:35.321690 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:45:36.295516 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:36.295479 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 18:45:36.295942 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:36.295532 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 18:45:37.295295 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:37.295249 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 23 18:45:37.295492 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:37.295307 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 23 18:45:38.298384 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:38.298353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:38.298773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:38.298631 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:38.299376 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:38.299360 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:45:55.231635 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.231593 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:55.232099 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.231861 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" podUID="9e429235-e0cf-420e-b944-9470d2354681" containerName="maas-api" containerID="cri-o://0727c37a98d7bbe3b9d3735200bc26ad75396f1bd9deea6952fcad118dee84fa" gracePeriod=30 Jul 23 18:45:55.447022 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.446986 2570 generic.go:356] "Generic (PLEG): container finished" podID="9e429235-e0cf-420e-b944-9470d2354681" containerID="0727c37a98d7bbe3b9d3735200bc26ad75396f1bd9deea6952fcad118dee84fa" exitCode=0 Jul 23 18:45:55.447220 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.447057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" event={"ID":"9e429235-e0cf-420e-b944-9470d2354681","Type":"ContainerDied","Data":"0727c37a98d7bbe3b9d3735200bc26ad75396f1bd9deea6952fcad118dee84fa"} Jul 23 18:45:55.485961 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.485870 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:55.586830 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.586801 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmvv\" (UniqueName: \"kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv\") pod \"9e429235-e0cf-420e-b944-9470d2354681\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " Jul 23 18:45:55.587032 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.586859 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") pod \"9e429235-e0cf-420e-b944-9470d2354681\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " Jul 23 18:45:55.587032 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.586910 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca\") pod \"9e429235-e0cf-420e-b944-9470d2354681\" (UID: \"9e429235-e0cf-420e-b944-9470d2354681\") " Jul 23 18:45:55.587328 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.587304 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e429235-e0cf-420e-b944-9470d2354681" (UID: "9e429235-e0cf-420e-b944-9470d2354681"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:45:55.589020 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.588999 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv" (OuterVolumeSpecName: "kube-api-access-4hmvv") pod "9e429235-e0cf-420e-b944-9470d2354681" (UID: "9e429235-e0cf-420e-b944-9470d2354681"). InnerVolumeSpecName "kube-api-access-4hmvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:45:55.589117 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.589023 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e429235-e0cf-420e-b944-9470d2354681" (UID: "9e429235-e0cf-420e-b944-9470d2354681"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:45:55.687996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.687953 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmvv\" (UniqueName: \"kubernetes.io/projected/9e429235-e0cf-420e-b944-9470d2354681-kube-api-access-4hmvv\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:55.687996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.687989 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e429235-e0cf-420e-b944-9470d2354681-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:55.687996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:55.688004 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e429235-e0cf-420e-b944-9470d2354681-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:45:56.453726 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:56.453699 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" Jul 23 18:45:56.454172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:56.453739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb" event={"ID":"9e429235-e0cf-420e-b944-9470d2354681","Type":"ContainerDied","Data":"f4f0a886a2ab709ecaf3a522709b4f8f9e63c5fa282999cd72f51313e2b9c189"} Jul 23 18:45:56.454172 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:56.453831 2570 scope.go:117] "RemoveContainer" containerID="0727c37a98d7bbe3b9d3735200bc26ad75396f1bd9deea6952fcad118dee84fa" Jul 23 18:45:56.481176 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:56.481125 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:56.486275 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:56.486243 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-dc11fe-5cd76b9b78-t7dqb"] Jul 23 18:45:57.142854 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:45:57.142821 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e429235-e0cf-420e-b944-9470d2354681" path="/var/lib/kubelet/pods/9e429235-e0cf-420e-b944-9470d2354681/volumes" Jul 23 18:46:07.447292 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:07.447252 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:46:07.447812 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:07.447502 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" containerID="cri-o://804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6" gracePeriod=30 Jul 23 18:46:08.301632 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:08.301597 2570 patch_prober.go:28] interesting pod/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:46:08.301908 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:08.301656 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:46:12.704905 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.704876 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:46:12.848558 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848520 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848583 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848607 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848624 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjlqg\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848672 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848713 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.848782 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848777 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.849106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848822 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.849106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848845 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data\") pod \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\" (UID: \"8d4b3b2f-75d8-4001-8e99-82cce84fb90c\") " Jul 23 18:46:12.849106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848956 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:46:12.849106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.848970 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:46:12.849326 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.849172 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.849326 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.849191 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.849326 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.849267 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:46:12.849461 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.849375 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:46:12.849719 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.849696 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:46:12.851026 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.851004 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg" (OuterVolumeSpecName: "kube-api-access-jjlqg") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "kube-api-access-jjlqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:46:12.851151 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.851114 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:46:12.851347 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.851329 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:46:12.851439 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.851340 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d4b3b2f-75d8-4001-8e99-82cce84fb90c" (UID: "8d4b3b2f-75d8-4001-8e99-82cce84fb90c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:46:12.950048 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950013 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlqg\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-kube-api-access-jjlqg\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950048 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950044 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950048 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950056 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950064 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950073 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950081 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:12.950276 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:12.950088 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4b3b2f-75d8-4001-8e99-82cce84fb90c-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:13.541778 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.541654 2570 generic.go:356] "Generic (PLEG): container finished" podID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerID="804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6" exitCode=0 Jul 23 18:46:13.541778 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.541700 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" Jul 23 18:46:13.541778 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.541765 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerDied","Data":"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6"} Jul 23 18:46:13.542077 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.541830 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq" event={"ID":"8d4b3b2f-75d8-4001-8e99-82cce84fb90c","Type":"ContainerDied","Data":"72a27323db7bb36b6d4681c0a10dcae999417d34c28c0e80dbc1615172d160d7"} Jul 23 18:46:13.542077 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.541859 2570 scope.go:117] "RemoveContainer" containerID="804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6" Jul 23 18:46:13.550684 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.550659 2570 scope.go:117] "RemoveContainer" containerID="e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e" Jul 23 18:46:13.559287 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.559261 2570 scope.go:117] "RemoveContainer" containerID="804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6" Jul 23 18:46:13.559629 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:13.559609 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6\": container with ID starting with 804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6 not found: ID does not exist" containerID="804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6" Jul 23 18:46:13.559690 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.559639 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6"} err="failed to get container status \"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6\": rpc error: code = NotFound desc = could not find container \"804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6\": container with ID starting with 804d33af37b950f16d2e3ad8499826263977ee12e2c5fad2fef58ee3823366b6 not found: ID does not exist" Jul 23 18:46:13.559690 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.559659 2570 scope.go:117] "RemoveContainer" containerID="e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e" Jul 23 18:46:13.559953 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:13.559931 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e\": container with ID starting with e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e not found: ID does not exist" containerID="e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e" Jul 23 18:46:13.560053 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.559956 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e"} err="failed to get container status \"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e\": rpc error: code = NotFound desc = could not find container \"e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e\": container with ID starting with e9b20106166fe0d7d9d092edbaed004e629f34ee78951898599c7b259d41797e not found: ID does not exist" Jul 23 18:46:13.563548 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.563521 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:46:13.573584 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:13.573547 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dc11fe-openshift-default-6fb4cb5b87-cw8rq"] Jul 23 18:46:15.142130 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:15.142096 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" path="/var/lib/kubelet/pods/8d4b3b2f-75d8-4001-8e99-82cce84fb90c/volumes" Jul 23 18:46:23.532236 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.532203 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:46:23.532683 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.532401 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" podUID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" containerName="maas-api" containerID="cri-o://3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f" gracePeriod=30 Jul 23 18:46:23.781801 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.781776 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:46:23.949460 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.949417 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") pod \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " Jul 23 18:46:23.949655 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.949571 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca\") pod \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " Jul 23 18:46:23.949655 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.949635 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk47f\" (UniqueName: \"kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f\") pod \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\" (UID: \"b3d78fbb-374c-47a5-9ee9-a9023adaa079\") " Jul 23 18:46:23.950014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.949986 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3d78fbb-374c-47a5-9ee9-a9023adaa079" (UID: "b3d78fbb-374c-47a5-9ee9-a9023adaa079"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:46:23.951651 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.951614 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f" (OuterVolumeSpecName: "kube-api-access-wk47f") pod "b3d78fbb-374c-47a5-9ee9-a9023adaa079" (UID: "b3d78fbb-374c-47a5-9ee9-a9023adaa079"). InnerVolumeSpecName "kube-api-access-wk47f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:46:23.951785 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:23.951706 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3d78fbb-374c-47a5-9ee9-a9023adaa079" (UID: "b3d78fbb-374c-47a5-9ee9-a9023adaa079"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:46:24.051310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.051270 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d78fbb-374c-47a5-9ee9-a9023adaa079-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:24.051310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.051304 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk47f\" (UniqueName: \"kubernetes.io/projected/b3d78fbb-374c-47a5-9ee9-a9023adaa079-kube-api-access-wk47f\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:24.051310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.051316 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d78fbb-374c-47a5-9ee9-a9023adaa079-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:46:24.596175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.596140 2570 generic.go:356] "Generic (PLEG): container finished" podID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" containerID="3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f" exitCode=0 Jul 23 18:46:24.596601 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.596202 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" Jul 23 18:46:24.596601 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.596233 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" event={"ID":"b3d78fbb-374c-47a5-9ee9-a9023adaa079","Type":"ContainerDied","Data":"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f"} Jul 23 18:46:24.596601 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.596291 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb" event={"ID":"b3d78fbb-374c-47a5-9ee9-a9023adaa079","Type":"ContainerDied","Data":"5055687321b88e1b0eed2e56e62dcf054410a15c08053ece5edc3227fa73fc30"} Jul 23 18:46:24.596601 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.596322 2570 scope.go:117] "RemoveContainer" containerID="3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f" Jul 23 18:46:24.604996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.604980 2570 scope.go:117] "RemoveContainer" containerID="3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f" Jul 23 18:46:24.605275 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:24.605255 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f\": container with ID starting with 3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f not found: ID does not exist" containerID="3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f" Jul 23 18:46:24.605316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.605284 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f"} err="failed to get container status \"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f\": rpc error: code = NotFound desc = could not find container \"3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f\": container with ID starting with 3c53bfc8b2bbc88636f530b834e31b165bccf6a5222146187013830982eddc4f not found: ID does not exist" Jul 23 18:46:24.624421 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.624390 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:46:24.631083 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:24.631054 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-81e4dc-56b85f499b-96dzb"] Jul 23 18:46:25.142198 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:25.142164 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" path="/var/lib/kubelet/pods/b3d78fbb-374c-47a5-9ee9-a9023adaa079/volumes" Jul 23 18:46:36.328124 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328076 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328389 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e429235-e0cf-420e-b944-9470d2354681" containerName="maas-api" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328401 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e429235-e0cf-420e-b944-9470d2354681" containerName="maas-api" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328417 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328422 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328438 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" containerName="maas-api" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328444 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" containerName="maas-api" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328493 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328503 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e429235-e0cf-420e-b944-9470d2354681" containerName="maas-api" Jul 23 18:46:36.328506 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.328509 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d78fbb-374c-47a5-9ee9-a9023adaa079" containerName="maas-api" Jul 23 18:46:36.331390 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.331372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.334289 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.334265 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-718e88-openshift-default-dockercfg-jfcfd\"" Jul 23 18:46:36.346361 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.346323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:46:36.454028 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.453984 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454223 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454223 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454142 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454223 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454167 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454374 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggk28\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454374 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454374 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454380 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.454475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.454410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555036 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.554997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggk28\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555054 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555468 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555614 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555807 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555787 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555897 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.555961 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555917 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.556021 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.555955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.556215 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.556192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.556535 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.556511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.557882 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.557862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.558021 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.557995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.565880 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.565856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.566658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.566636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggk28\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28\") pod \"e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.640972 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.640870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:36.785268 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.785236 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:46:36.787844 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:46:36.787807 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046c1568_1b24_4751_b549_f6dd5744ee39.slice/crio-266e5c9caf558278267b0b54294a7984bf8ef28661e18b56c1283a725b2e8727 WatchSource:0}: Error finding container 266e5c9caf558278267b0b54294a7984bf8ef28661e18b56c1283a725b2e8727: Status 404 returned error can't find the container with id 266e5c9caf558278267b0b54294a7984bf8ef28661e18b56c1283a725b2e8727 Jul 23 18:46:36.789861 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.789829 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:36.789947 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.789899 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:36.789947 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:36.789925 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:37.206667 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:37.206636 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:46:37.212006 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:37.211986 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:46:37.664550 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:37.664514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerStarted","Data":"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed"} Jul 23 18:46:37.664956 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:37.664568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerStarted","Data":"266e5c9caf558278267b0b54294a7984bf8ef28661e18b56c1283a725b2e8727"} Jul 23 18:46:37.692109 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:37.692058 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podStartSLOduration=1.692039269 podStartE2EDuration="1.692039269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:46:37.6895566 +0000 UTC m=+2401.257153786" watchObservedRunningTime="2026-07-23 18:46:37.692039269 +0000 UTC m=+2401.259636442" Jul 23 18:46:38.641728 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:38.641681 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:38.646584 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:38.646560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:38.669059 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:38.669034 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:38.670196 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:38.670179 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:39.918148 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.918114 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:46:39.918530 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.918450 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:39.918530 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.918461 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:39.918530 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.918516 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4b3b2f-75d8-4001-8e99-82cce84fb90c" containerName="istio-proxy" Jul 23 18:46:39.923169 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.923137 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:39.927129 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.927091 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-718e88-serving-cert\"" Jul 23 18:46:39.933836 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.933801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:46:39.982088 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.982047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:39.982282 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.982106 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:39.982282 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:39.982144 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp96m\" (UniqueName: \"kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.082722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.082683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.082945 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.082739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp96m\" (UniqueName: \"kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.082945 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.082828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.082945 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:40.082871 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-serving-cert: secret "maas-api-e2e-shared-a-718e88-serving-cert" not found Jul 23 18:46:40.083117 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:40.082967 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls podName:66acfec9-4c94-4bf0-9cf9-c688f6ff94ee nodeName:}" failed. No retries permitted until 2026-07-23 18:46:40.582943471 +0000 UTC m=+2404.150540621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls") pod "maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" (UID: "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee") : secret "maas-api-e2e-shared-a-718e88-serving-cert" not found Jul 23 18:46:40.083577 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.083559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.107340 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.107313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp96m\" (UniqueName: \"kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.587822 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.587785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.590334 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.590293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") pod \"maas-api-e2e-shared-a-718e88-69587b478b-rkjcb\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.840233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.840134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:40.980605 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:40.980506 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:46:40.983105 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:46:40.983074 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66acfec9_4c94_4bf0_9cf9_c688f6ff94ee.slice/crio-d768871d99cf593ffdf3fb0745c527b710647cbe25086b744b4b68200e6dd2c1 WatchSource:0}: Error finding container d768871d99cf593ffdf3fb0745c527b710647cbe25086b744b4b68200e6dd2c1: Status 404 returned error can't find the container with id d768871d99cf593ffdf3fb0745c527b710647cbe25086b744b4b68200e6dd2c1 Jul 23 18:46:41.686650 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:41.686611 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" event={"ID":"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee","Type":"ContainerStarted","Data":"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008"} Jul 23 18:46:41.686852 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:41.686667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" event={"ID":"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee","Type":"ContainerStarted","Data":"d768871d99cf593ffdf3fb0745c527b710647cbe25086b744b4b68200e6dd2c1"} Jul 23 18:46:41.686852 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:41.686694 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:41.706760 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:41.706638 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" podStartSLOduration=2.24394809 podStartE2EDuration="2.706621509s" totalImagesPullingTime="462.673419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:46:41.704658404 +0000 UTC m=+2405.272255596" watchObservedRunningTime="2026-07-23 18:46:41.706621509 +0000 UTC m=+2405.274218679" Jul 23 18:46:47.698954 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:47.698925 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:46:48.057557 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.057466 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:46:48.060282 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.060260 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.063059 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.063042 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2fc9f5-serving-cert\"" Jul 23 18:46:48.078093 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.078067 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:46:48.156336 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.156299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.156554 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.156362 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.156554 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.156532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5hd\" (UniqueName: \"kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.257501 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.257456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5hd\" (UniqueName: \"kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.258206 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.258183 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.258508 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.258490 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.259495 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.259473 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.259654 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:48.258430 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-serving-cert: secret "maas-api-e2e-shared-b-2fc9f5-serving-cert" not found Jul 23 18:46:48.259825 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:46:48.259814 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls podName:a83dbde7-826b-4f07-b098-b167d7a51f5d nodeName:}" failed. No retries permitted until 2026-07-23 18:46:48.759791772 +0000 UTC m=+2412.327388936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls") pod "maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" (UID: "a83dbde7-826b-4f07-b098-b167d7a51f5d") : secret "maas-api-e2e-shared-b-2fc9f5-serving-cert" not found Jul 23 18:46:48.269966 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.269902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5hd\" (UniqueName: \"kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.764813 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.764129 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.768930 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.768863 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:48.971296 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:48.970985 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:49.143175 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:49.143149 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:46:49.143872 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:46:49.143843 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83dbde7_826b_4f07_b098_b167d7a51f5d.slice/crio-cc4ffeaeb81912a957608af58185b80cd0f51f9b705d9724ed566ecae3b0adc6 WatchSource:0}: Error finding container cc4ffeaeb81912a957608af58185b80cd0f51f9b705d9724ed566ecae3b0adc6: Status 404 returned error can't find the container with id cc4ffeaeb81912a957608af58185b80cd0f51f9b705d9724ed566ecae3b0adc6 Jul 23 18:46:49.729106 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:49.729065 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" event={"ID":"a83dbde7-826b-4f07-b098-b167d7a51f5d","Type":"ContainerStarted","Data":"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf"} Jul 23 18:46:49.729288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:49.729130 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" event={"ID":"a83dbde7-826b-4f07-b098-b167d7a51f5d","Type":"ContainerStarted","Data":"cc4ffeaeb81912a957608af58185b80cd0f51f9b705d9724ed566ecae3b0adc6"} Jul 23 18:46:49.729288 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:49.729211 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:49.752224 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:49.752151 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" podStartSLOduration=1.31722481 podStartE2EDuration="1.752128613s" totalImagesPullingTime="434.903803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:46:49.74906797 +0000 UTC m=+2413.316665144" watchObservedRunningTime="2026-07-23 18:46:49.752128613 +0000 UTC m=+2413.319725788" Jul 23 18:46:54.710996 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:54.710953 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 18:46:54.711426 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:54.711028 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 18:46:55.740167 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.740138 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:46:55.765298 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.765255 2570 generic.go:356] "Generic (PLEG): container finished" podID="046c1568-1b24-4751-b549-f6dd5744ee39" containerID="2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed" exitCode=0 Jul 23 18:46:55.765472 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.765329 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerDied","Data":"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed"} Jul 23 18:46:55.765780 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.765765 2570 scope.go:117] "RemoveContainer" containerID="2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed" Jul 23 18:46:55.766708 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.766682 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:55.766829 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.766762 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:55.766829 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:55.766794 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 18:46:56.776314 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:56.776267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerStarted","Data":"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2"} Jul 23 18:46:57.641836 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:57.641792 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:46:57.642107 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:57.642083 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:46:57.642163 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:57.642135 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 23 18:46:58.641834 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:58.641730 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:46:58.641834 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:58.641812 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 23 18:46:59.642107 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:59.642067 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:46:59.642523 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:46:59.642138 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 23 18:47:00.641375 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:47:00.641327 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 23 18:47:00.641614 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:47:00.641395 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 23 18:47:01.645387 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:47:01.645351 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:47:01.645784 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:47:01.645623 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:47:01.646350 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:47:01.646333 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:49:37.524805 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:37.524776 2570 scope.go:117] "RemoveContainer" containerID="ecf8d59527318368505c95219789ab6de2f61c1dbda2456c7a46763b5943f5eb" Jul 23 18:49:40.891858 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.891821 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:49:40.896351 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.896327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:40.901481 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.901458 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-718e88\"/\"kube-root-ca.crt\"" Jul 23 18:49:40.902497 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.902464 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-718e88\"/\"openshift-service-ca.crt\"" Jul 23 18:49:40.902644 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.902522 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-718e88\"/\"default-dockercfg-8rxbv\"" Jul 23 18:49:40.902644 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.902561 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-718e88\"/\"sub-test-model-718e88-kserve-self-signed-certs\"" Jul 23 18:49:40.908310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:40.908281 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:49:41.004809 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.004739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.004993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.004851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.004993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.004869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.004993 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.004903 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.005108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.004985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.005108 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.005015 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v6j\" (UniqueName: \"kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106378 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106470 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v6j\" (UniqueName: \"kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.106942 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106915 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.107063 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.106962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.107063 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.107015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.108816 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.108796 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.109199 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.109178 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.118321 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.118292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v6j\" (UniqueName: \"kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j\") pod \"sub-test-model-718e88-kserve-8cb9ff6df-2mzqg\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.205475 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.205365 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:41.347560 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.347532 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:49:41.350251 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:49:41.350221 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6afdcf_93cd_40e8_b024_60075a7791d8.slice/crio-34e98155fd6a5644d5e09b0e626ff160ed561154789fae98558d3bd10808847f WatchSource:0}: Error finding container 34e98155fd6a5644d5e09b0e626ff160ed561154789fae98558d3bd10808847f: Status 404 returned error can't find the container with id 34e98155fd6a5644d5e09b0e626ff160ed561154789fae98558d3bd10808847f Jul 23 18:49:41.609667 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.609634 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" event={"ID":"ee6afdcf-93cd-40e8-b024-60075a7791d8","Type":"ContainerStarted","Data":"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f"} Jul 23 18:49:41.609943 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.609673 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" event={"ID":"ee6afdcf-93cd-40e8-b024-60075a7791d8","Type":"ContainerStarted","Data":"34e98155fd6a5644d5e09b0e626ff160ed561154789fae98558d3bd10808847f"} Jul 23 18:49:41.632578 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:41.632515 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" podStartSLOduration=1.632494526 podStartE2EDuration="1.632494526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 18:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:49:41.631424368 +0000 UTC m=+2585.199021538" watchObservedRunningTime="2026-07-23 18:49:41.632494526 +0000 UTC m=+2585.200091697" Jul 23 18:49:51.206095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:51.206047 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:51.206095 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:51.206099 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:51.218793 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:51.218762 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:49:51.670188 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:49:51.670158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:50:56.988502 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:50:56.988456 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:51:12.008514 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.008457 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" podUID="ee6afdcf-93cd-40e8-b024-60075a7791d8" containerName="main" containerID="cri-o://a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f" gracePeriod=45 Jul 23 18:51:12.257191 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.257165 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:51:12.332078 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332046 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332114 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332141 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332180 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5v6j\" (UniqueName: \"kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332278 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs\") pod \"ee6afdcf-93cd-40e8-b024-60075a7791d8\" (UID: \"ee6afdcf-93cd-40e8-b024-60075a7791d8\") " Jul 23 18:51:12.332487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332337 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home" (OuterVolumeSpecName: "home") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:51:12.332487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332370 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:51:12.332487 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332448 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:51:12.332642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332550 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-home\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:12.332642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332561 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-model-cache\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:12.332642 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.332571 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-tmp-dir\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:12.334458 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.334438 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm" (OuterVolumeSpecName: "dshm") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:51:12.334540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.334447 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j" (OuterVolumeSpecName: "kube-api-access-t5v6j") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "kube-api-access-t5v6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:51:12.334540 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.334441 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee6afdcf-93cd-40e8-b024-60075a7791d8" (UID: "ee6afdcf-93cd-40e8-b024-60075a7791d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:51:12.433491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.433452 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6afdcf-93cd-40e8-b024-60075a7791d8-tls-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:12.433491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.433483 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6afdcf-93cd-40e8-b024-60075a7791d8-dshm\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:12.433491 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:12.433492 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5v6j\" (UniqueName: \"kubernetes.io/projected/ee6afdcf-93cd-40e8-b024-60075a7791d8-kube-api-access-t5v6j\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:51:13.058545 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.058507 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee6afdcf-93cd-40e8-b024-60075a7791d8" containerID="a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f" exitCode=0 Jul 23 18:51:13.058987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.058548 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" Jul 23 18:51:13.058987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.058595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" event={"ID":"ee6afdcf-93cd-40e8-b024-60075a7791d8","Type":"ContainerDied","Data":"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f"} Jul 23 18:51:13.058987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.058637 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg" event={"ID":"ee6afdcf-93cd-40e8-b024-60075a7791d8","Type":"ContainerDied","Data":"34e98155fd6a5644d5e09b0e626ff160ed561154789fae98558d3bd10808847f"} Jul 23 18:51:13.058987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.058656 2570 scope.go:117] "RemoveContainer" containerID="a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f" Jul 23 18:51:13.067920 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.067892 2570 scope.go:117] "RemoveContainer" containerID="a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f" Jul 23 18:51:13.068183 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:51:13.068163 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f\": container with ID starting with a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f not found: ID does not exist" containerID="a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f" Jul 23 18:51:13.068246 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.068200 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f"} err="failed to get container status \"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f\": rpc error: code = NotFound desc = could not find container \"a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f\": container with ID starting with a9044a1a71738c6b26504a81a4fb311108612a8f75c3c7bbea61a6c64e33da9f not found: ID does not exist" Jul 23 18:51:13.081659 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.081622 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:51:13.086109 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.086083 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-718e88/sub-test-model-718e88-kserve-8cb9ff6df-2mzqg"] Jul 23 18:51:13.143301 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:13.143268 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee6afdcf-93cd-40e8-b024-60075a7791d8" path="/var/lib/kubelet/pods/ee6afdcf-93cd-40e8-b024-60075a7791d8/volumes" Jul 23 18:51:37.237710 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:37.237685 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:51:37.243267 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:51:37.243244 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vwjpk_9cce2639-0478-49ca-a434-208bfc9dd501/ovn-acl-logging/0.log" Jul 23 18:53:02.910203 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:02.910100 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:53:02.910779 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:02.910435 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" podUID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" containerName="maas-api" containerID="cri-o://b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008" gracePeriod=30 Jul 23 18:53:04.155295 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.155271 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:53:04.333507 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.333474 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") pod \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " Jul 23 18:53:04.333690 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.333583 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca\") pod \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " Jul 23 18:53:04.333690 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.333618 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp96m\" (UniqueName: \"kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m\") pod \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\" (UID: \"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee\") " Jul 23 18:53:04.334016 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.333982 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" (UID: "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:53:04.335728 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.335699 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" (UID: "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:53:04.335862 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.335764 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m" (OuterVolumeSpecName: "kube-api-access-tp96m") pod "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" (UID: "66acfec9-4c94-4bf0-9cf9-c688f6ff94ee"). InnerVolumeSpecName "kube-api-access-tp96m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:53:04.434884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.434849 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp96m\" (UniqueName: \"kubernetes.io/projected/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-kube-api-access-tp96m\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:04.434884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.434879 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:04.434884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.434888 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:04.605674 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.605576 2570 generic.go:356] "Generic (PLEG): container finished" podID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" containerID="b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008" exitCode=0 Jul 23 18:53:04.605674 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.605648 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" Jul 23 18:53:04.605959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.605666 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" event={"ID":"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee","Type":"ContainerDied","Data":"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008"} Jul 23 18:53:04.605959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.605782 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb" event={"ID":"66acfec9-4c94-4bf0-9cf9-c688f6ff94ee","Type":"ContainerDied","Data":"d768871d99cf593ffdf3fb0745c527b710647cbe25086b744b4b68200e6dd2c1"} Jul 23 18:53:04.605959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.605819 2570 scope.go:117] "RemoveContainer" containerID="b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008" Jul 23 18:53:04.614736 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.614716 2570 scope.go:117] "RemoveContainer" containerID="b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008" Jul 23 18:53:04.615052 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:53:04.615030 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008\": container with ID starting with b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008 not found: ID does not exist" containerID="b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008" Jul 23 18:53:04.615101 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.615061 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008"} err="failed to get container status \"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008\": rpc error: code = NotFound desc = could not find container \"b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008\": container with ID starting with b1927d2703521ce603e9d132ce5ecebb1ab925d2e46aa3ee1ce583061e410008 not found: ID does not exist" Jul 23 18:53:04.640842 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.640807 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:53:04.648182 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:04.648154 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-718e88-69587b478b-rkjcb"] Jul 23 18:53:05.142193 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:05.142155 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" path="/var/lib/kubelet/pods/66acfec9-4c94-4bf0-9cf9-c688f6ff94ee/volumes" Jul 23 18:53:14.889238 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:14.889193 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:53:14.889858 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:14.889464 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" containerID="cri-o://0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2" gracePeriod=30 Jul 23 18:53:16.647727 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:16.647687 2570 patch_prober.go:28] interesting pod/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 18:53:16.648138 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:16.647780 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 18:53:20.141793 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.141764 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:53:20.179555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179472 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179514 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179555 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179543 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179577 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179616 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggk28\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179649 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179683 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179730 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179785 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179811 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo\") pod \"046c1568-1b24-4751-b549-f6dd5744ee39\" (UID: \"046c1568-1b24-4751-b549-f6dd5744ee39\") " Jul 23 18:53:20.179959 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.179840 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:53:20.180364 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.180044 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:53:20.180364 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.180079 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.180364 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.180098 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-credential-socket\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.180364 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.180217 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:53:20.180535 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.180510 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data" (OuterVolumeSpecName: "istio-data") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:53:20.182027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.182002 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28" (OuterVolumeSpecName: "kube-api-access-ggk28") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "kube-api-access-ggk28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:53:20.182279 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.182259 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 18:53:20.182423 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.182407 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 18:53:20.182617 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.182599 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token" (OuterVolumeSpecName: "istio-token") pod "046c1568-1b24-4751-b549-f6dd5744ee39" (UID: "046c1568-1b24-4751-b549-f6dd5744ee39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:53:20.281057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281022 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-istio-token\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281051 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/046c1568-1b24-4751-b549-f6dd5744ee39-istio-podinfo\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281060 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-data\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281057 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281068 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/046c1568-1b24-4751-b549-f6dd5744ee39-istiod-ca-cert\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281339 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281078 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggk28\" (UniqueName: \"kubernetes.io/projected/046c1568-1b24-4751-b549-f6dd5744ee39-kube-api-access-ggk28\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281339 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281086 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-workload-certs\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.281339 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.281094 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/046c1568-1b24-4751-b549-f6dd5744ee39-istio-envoy\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:20.686177 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.686136 2570 generic.go:356] "Generic (PLEG): container finished" podID="046c1568-1b24-4751-b549-f6dd5744ee39" containerID="0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2" exitCode=0 Jul 23 18:53:20.686386 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.686201 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" Jul 23 18:53:20.686386 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.686224 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerDied","Data":"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2"} Jul 23 18:53:20.686386 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.686287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm" event={"ID":"046c1568-1b24-4751-b549-f6dd5744ee39","Type":"ContainerDied","Data":"266e5c9caf558278267b0b54294a7984bf8ef28661e18b56c1283a725b2e8727"} Jul 23 18:53:20.686386 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.686319 2570 scope.go:117] "RemoveContainer" containerID="0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2" Jul 23 18:53:20.695849 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.695830 2570 scope.go:117] "RemoveContainer" containerID="2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed" Jul 23 18:53:20.704316 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.704294 2570 scope.go:117] "RemoveContainer" containerID="0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2" Jul 23 18:53:20.704620 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:53:20.704598 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2\": container with ID starting with 0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2 not found: ID does not exist" containerID="0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2" Jul 23 18:53:20.704687 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.704628 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2"} err="failed to get container status \"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2\": rpc error: code = NotFound desc = could not find container \"0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2\": container with ID starting with 0cf58d67e0289fc9c0293c6485d97e9ed81946f66b820b1d35dbb6cba8ef42b2 not found: ID does not exist" Jul 23 18:53:20.704687 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.704650 2570 scope.go:117] "RemoveContainer" containerID="2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed" Jul 23 18:53:20.704931 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:53:20.704913 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed\": container with ID starting with 2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed not found: ID does not exist" containerID="2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed" Jul 23 18:53:20.704987 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.704937 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed"} err="failed to get container status \"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed\": rpc error: code = NotFound desc = could not find container \"2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed\": container with ID starting with 2832b08b8d3f4325f4a4ab6d3dec59b0711f89d27629ce0e6106f2a9575581ed not found: ID does not exist" Jul 23 18:53:20.710105 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.710076 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:53:20.715082 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:20.715053 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-718e88-openshift-default-795cb8db67-4j5dm"] Jul 23 18:53:21.142357 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:21.142324 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" path="/var/lib/kubelet/pods/046c1568-1b24-4751-b549-f6dd5744ee39/volumes" Jul 23 18:53:25.894090 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:25.894052 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:53:25.894562 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:25.894261 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" podUID="a83dbde7-826b-4f07-b098-b167d7a51f5d" containerName="maas-api" containerID="cri-o://12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf" gracePeriod=30 Jul 23 18:53:26.141752 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.141713 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:53:26.231784 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.231654 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") pod \"a83dbde7-826b-4f07-b098-b167d7a51f5d\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " Jul 23 18:53:26.231784 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.231698 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca\") pod \"a83dbde7-826b-4f07-b098-b167d7a51f5d\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " Jul 23 18:53:26.232030 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.231798 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx5hd\" (UniqueName: \"kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd\") pod \"a83dbde7-826b-4f07-b098-b167d7a51f5d\" (UID: \"a83dbde7-826b-4f07-b098-b167d7a51f5d\") " Jul 23 18:53:26.232147 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.232119 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a83dbde7-826b-4f07-b098-b167d7a51f5d" (UID: "a83dbde7-826b-4f07-b098-b167d7a51f5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 18:53:26.233875 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.233850 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a83dbde7-826b-4f07-b098-b167d7a51f5d" (UID: "a83dbde7-826b-4f07-b098-b167d7a51f5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 18:53:26.233989 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.233940 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd" (OuterVolumeSpecName: "kube-api-access-hx5hd") pod "a83dbde7-826b-4f07-b098-b167d7a51f5d" (UID: "a83dbde7-826b-4f07-b098-b167d7a51f5d"). InnerVolumeSpecName "kube-api-access-hx5hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 18:53:26.333006 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.332957 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5hd\" (UniqueName: \"kubernetes.io/projected/a83dbde7-826b-4f07-b098-b167d7a51f5d-kube-api-access-hx5hd\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:26.333006 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.332996 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a83dbde7-826b-4f07-b098-b167d7a51f5d-maas-api-tls\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:26.333006 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.333006 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83dbde7-826b-4f07-b098-b167d7a51f5d-openshift-service-ca\") on node \"ip-10-0-132-197.ec2.internal\" DevicePath \"\"" Jul 23 18:53:26.719791 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.719739 2570 generic.go:356] "Generic (PLEG): container finished" podID="a83dbde7-826b-4f07-b098-b167d7a51f5d" containerID="12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf" exitCode=0 Jul 23 18:53:26.720014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.719801 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" Jul 23 18:53:26.720014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.719807 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" event={"ID":"a83dbde7-826b-4f07-b098-b167d7a51f5d","Type":"ContainerDied","Data":"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf"} Jul 23 18:53:26.720014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.719950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh" event={"ID":"a83dbde7-826b-4f07-b098-b167d7a51f5d","Type":"ContainerDied","Data":"cc4ffeaeb81912a957608af58185b80cd0f51f9b705d9724ed566ecae3b0adc6"} Jul 23 18:53:26.720014 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.719988 2570 scope.go:117] "RemoveContainer" containerID="12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf" Jul 23 18:53:26.729490 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.729469 2570 scope.go:117] "RemoveContainer" containerID="12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf" Jul 23 18:53:26.729804 ip-10-0-132-197 kubenswrapper[2570]: E0723 18:53:26.729782 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf\": container with ID starting with 12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf not found: ID does not exist" containerID="12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf" Jul 23 18:53:26.729885 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.729817 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf"} err="failed to get container status \"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf\": rpc error: code = NotFound desc = could not find container \"12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf\": container with ID starting with 12254c03b2c02a6999705ad538d522c85a007b300dd8ac23466a12326511ebbf not found: ID does not exist" Jul 23 18:53:26.744060 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.744022 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:53:26.750399 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:26.750370 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2fc9f5-5d5b45b87f-ssgqh"] Jul 23 18:53:27.142002 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:27.141969 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83dbde7-826b-4f07-b098-b167d7a51f5d" path="/var/lib/kubelet/pods/a83dbde7-826b-4f07-b098-b167d7a51f5d/volumes" Jul 23 18:53:46.295569 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:46.295533 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f469bfccf-8stxv_c8c15c67-a4f6-4d17-afd2-785ec6019412/manager/0.log" Jul 23 18:53:47.018594 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:47.018567 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b5877c86-zg6g4_c0ed3ee6-a6f1-4841-b04c-df081ff26e1c/copy-manifests/0.log" Jul 23 18:53:47.028707 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:47.028670 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b5877c86-zg6g4_c0ed3ee6-a6f1-4841-b04c-df081ff26e1c/manager/0.log" Jul 23 18:53:47.415521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:47.415484 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-678f786967-bhbf4_2660ffc3-0a63-4d50-b75f-6a14745c9f46/maas-api/0.log" Jul 23 18:53:48.646988 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.646952 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/util/0.log" Jul 23 18:53:48.664955 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.664932 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/pull/0.log" Jul 23 18:53:48.677088 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.677057 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/extract/0.log" Jul 23 18:53:48.820762 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.820718 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/util/0.log" Jul 23 18:53:48.844437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.844412 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/pull/0.log" Jul 23 18:53:48.855866 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.855838 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/extract/0.log" Jul 23 18:53:48.982046 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.981969 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/util/0.log" Jul 23 18:53:48.989992 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.989964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/pull/0.log" Jul 23 18:53:48.998820 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:48.998772 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/extract/0.log" Jul 23 18:53:49.110286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:49.110254 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/pull/0.log" Jul 23 18:53:49.117810 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:49.117782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/extract/0.log" Jul 23 18:53:49.125658 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:49.125627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/util/0.log" Jul 23 18:53:50.530188 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:50.530158 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678z247t_69a96141-813c-41c7-bac8-54834b9fcddb/istio-proxy/0.log" Jul 23 18:53:50.671949 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:50.671916 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94br4_c25c8216-25cc-4161-aca2-9f98a47ecec8/discovery/0.log" Jul 23 18:53:51.036453 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:51.036416 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/1.log" Jul 23 18:53:51.284755 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:51.284711 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-c29g8_d66b6946-2d91-426e-8466-24f5075a9d8d/payload-processing/0.log" Jul 23 18:53:52.000492 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:53:52.000450 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-v8lqw_1b068fb1-045b-4e5b-920e-7d52a14ccd55/main/0.log" Jul 23 18:54:01.853437 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:01.853393 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bmskp_b766b43e-8eb4-43bd-9671-ed7c0c3a0e98/global-pull-secret-syncer/0.log" Jul 23 18:54:02.135884 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:02.135804 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bx84d_5c8d624b-754c-45c4-98f6-848d0fc2f37b/konnectivity-agent/0.log" Jul 23 18:54:02.258070 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:02.258041 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-197.ec2.internal_bd05e8e976bdb21bacf0ae5f21b68be6/haproxy/0.log" Jul 23 18:54:06.465521 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.465469 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/extract/0.log" Jul 23 18:54:06.577872 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.577839 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/util/0.log" Jul 23 18:54:06.612536 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.612506 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hcpqs_4405a7be-7b7d-46ef-b6c8-81a9b2ac9358/pull/0.log" Jul 23 18:54:06.644817 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.644787 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/extract/0.log" Jul 23 18:54:06.669636 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.669604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/util/0.log" Jul 23 18:54:06.697489 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.697465 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e027gdq_50eb55f1-f0dd-4954-bdc7-86c45b2505e0/pull/0.log" Jul 23 18:54:06.728649 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.728556 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/extract/0.log" Jul 23 18:54:06.753290 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.753265 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/util/0.log" Jul 23 18:54:06.785989 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.785956 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7359gpb_2bd114a6-b51f-457c-b7a2-5ddb3782af9c/pull/0.log" Jul 23 18:54:06.821262 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.821226 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/extract/0.log" Jul 23 18:54:06.846605 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.846579 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/util/0.log" Jul 23 18:54:06.872713 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:06.872685 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1phtpg_114801d9-2781-4dd0-b2a2-2dea008f50a6/pull/0.log" Jul 23 18:54:10.106722 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.106693 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zbbx_baa81db5-e393-4625-b648-cd2e5da1566d/node-exporter/0.log" Jul 23 18:54:10.133861 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.133834 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zbbx_baa81db5-e393-4625-b648-cd2e5da1566d/kube-rbac-proxy/0.log" Jul 23 18:54:10.160583 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.160550 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zbbx_baa81db5-e393-4625-b648-cd2e5da1566d/init-textfile/0.log" Jul 23 18:54:10.380933 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.380840 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5tddh_b228f96e-2f05-451d-8230-0c11be457cb4/kube-rbac-proxy-main/0.log" Jul 23 18:54:10.410229 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.410197 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5tddh_b228f96e-2f05-451d-8230-0c11be457cb4/kube-rbac-proxy-self/0.log" Jul 23 18:54:10.436714 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:10.436686 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-5tddh_b228f96e-2f05-451d-8230-0c11be457cb4/openshift-state-metrics/0.log" Jul 23 18:54:11.710365 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710328 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx"] Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710615 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710625 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710633 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83dbde7-826b-4f07-b098-b167d7a51f5d" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710639 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83dbde7-826b-4f07-b098-b167d7a51f5d" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710650 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee6afdcf-93cd-40e8-b024-60075a7791d8" containerName="main" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710656 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee6afdcf-93cd-40e8-b024-60075a7791d8" containerName="main" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710664 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710669 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710736 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="66acfec9-4c94-4bf0-9cf9-c688f6ff94ee" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710759 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83dbde7-826b-4f07-b098-b167d7a51f5d" containerName="maas-api" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710769 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710775 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee6afdcf-93cd-40e8-b024-60075a7791d8" containerName="main" Jul 23 18:54:11.710773 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.710781 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="046c1568-1b24-4751-b549-f6dd5744ee39" containerName="istio-proxy" Jul 23 18:54:11.714182 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.714161 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.716731 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.716701 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc7fz\"/\"openshift-service-ca.crt\"" Jul 23 18:54:11.716731 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.716704 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc7fz\"/\"kube-root-ca.crt\"" Jul 23 18:54:11.717769 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.717738 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jc7fz\"/\"default-dockercfg-zx572\"" Jul 23 18:54:11.723735 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.723713 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx"] Jul 23 18:54:11.829486 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.829451 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsgzk\" (UniqueName: \"kubernetes.io/projected/b215e365-99ea-467d-ad78-d56817f7d129-kube-api-access-qsgzk\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.829693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.829509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-sys\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.829693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.829535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-proc\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.829693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.829568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-podres\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.829693 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.829588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-lib-modules\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930765 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsgzk\" (UniqueName: \"kubernetes.io/projected/b215e365-99ea-467d-ad78-d56817f7d129-kube-api-access-qsgzk\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930816 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-sys\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930840 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-proc\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-podres\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-lib-modules\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930960 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-proc\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.930981 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.930944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-sys\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.931233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.931001 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-podres\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.931233 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.931035 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b215e365-99ea-467d-ad78-d56817f7d129-lib-modules\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:11.941667 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:11.941632 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsgzk\" (UniqueName: \"kubernetes.io/projected/b215e365-99ea-467d-ad78-d56817f7d129-kube-api-access-qsgzk\") pod \"perf-node-gather-daemonset-7j2cx\" (UID: \"b215e365-99ea-467d-ad78-d56817f7d129\") " pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:12.024559 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.024472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:12.165871 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.165843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx"] Jul 23 18:54:12.167954 ip-10-0-132-197 kubenswrapper[2570]: W0723 18:54:12.167925 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb215e365_99ea_467d_ad78_d56817f7d129.slice/crio-6969db55a5b629c61ee2cb4c6baf50d2cb2756b99228d2bcf22960c477b28497 WatchSource:0}: Error finding container 6969db55a5b629c61ee2cb4c6baf50d2cb2756b99228d2bcf22960c477b28497: Status 404 returned error can't find the container with id 6969db55a5b629c61ee2cb4c6baf50d2cb2756b99228d2bcf22960c477b28497 Jul 23 18:54:12.941148 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.941110 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" event={"ID":"b215e365-99ea-467d-ad78-d56817f7d129","Type":"ContainerStarted","Data":"e3f18f12a353585c8b4e2467c029fde9911b0e584b19d069eb2e6b20728d7d16"} Jul 23 18:54:12.941548 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.941167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" event={"ID":"b215e365-99ea-467d-ad78-d56817f7d129","Type":"ContainerStarted","Data":"6969db55a5b629c61ee2cb4c6baf50d2cb2756b99228d2bcf22960c477b28497"} Jul 23 18:54:12.941548 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.941193 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:12.961698 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:12.961649 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" podStartSLOduration=1.961633257 podStartE2EDuration="1.961633257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 18:54:12.959794506 +0000 UTC m=+2856.527391677" watchObservedRunningTime="2026-07-23 18:54:12.961633257 +0000 UTC m=+2856.529230427" Jul 23 18:54:15.489571 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:15.489541 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-smfl5_cb0701e9-aee0-41c9-9900-fc2d46bfd966/dns/0.log" Jul 23 18:54:15.520937 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:15.520910 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-smfl5_cb0701e9-aee0-41c9-9900-fc2d46bfd966/kube-rbac-proxy/0.log" Jul 23 18:54:15.584638 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:15.584614 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5fs28_6f94a6d3-48dc-4cee-8af9-f8585b8fb973/dns-node-resolver/0.log" Jul 23 18:54:16.307632 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:16.307604 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbsxq_2e4b3d0d-f43f-46a1-a999-440f0985c9ac/node-ca/0.log" Jul 23 18:54:17.532739 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:17.532700 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678z247t_69a96141-813c-41c7-bac8-54834b9fcddb/istio-proxy/0.log" Jul 23 18:54:17.766526 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:17.766494 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94br4_c25c8216-25cc-4161-aca2-9f98a47ecec8/discovery/0.log" Jul 23 18:54:17.881901 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:17.881871 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/0.log" Jul 23 18:54:17.932310 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:17.932258 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/1.log" Jul 23 18:54:18.011939 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:18.011905 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-c29g8_d66b6946-2d91-426e-8466-24f5075a9d8d/payload-processing/0.log" Jul 23 18:54:18.870277 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:18.870246 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l8sxs_2cee312f-0de4-436b-bd3c-21e0ed6f93d9/serve-healthcheck-canary/0.log" Jul 23 18:54:18.956062 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:18.956032 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jc7fz/perf-node-gather-daemonset-7j2cx" Jul 23 18:54:19.407396 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:19.407367 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v9vc_f1ac5912-aaf9-4eb0-8212-fb607a65fdc9/kube-rbac-proxy/0.log" Jul 23 18:54:19.435479 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:19.435449 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v9vc_f1ac5912-aaf9-4eb0-8212-fb607a65fdc9/exporter/0.log" Jul 23 18:54:19.463035 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:19.463000 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5v9vc_f1ac5912-aaf9-4eb0-8212-fb607a65fdc9/extractor/0.log" Jul 23 18:54:22.710676 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:22.710627 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f469bfccf-8stxv_c8c15c67-a4f6-4d17-afd2-785ec6019412/manager/0.log" Jul 23 18:54:22.853324 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:22.853295 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678z247t_69a96141-813c-41c7-bac8-54834b9fcddb/istio-proxy/0.log" Jul 23 18:54:23.077103 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.077065 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94br4_c25c8216-25cc-4161-aca2-9f98a47ecec8/discovery/0.log" Jul 23 18:54:23.107129 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.107100 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b5877c86-zg6g4_c0ed3ee6-a6f1-4841-b04c-df081ff26e1c/manager/0.log" Jul 23 18:54:23.132698 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.132658 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/0.log" Jul 23 18:54:23.137266 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.137234 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56b5877c86-zg6g4_c0ed3ee6-a6f1-4841-b04c-df081ff26e1c/copy-manifests/0.log" Jul 23 18:54:23.188027 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.187989 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/1.log" Jul 23 18:54:23.246517 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:23.246488 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-c29g8_d66b6946-2d91-426e-8466-24f5075a9d8d/payload-processing/0.log" Jul 23 18:54:24.484286 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.484257 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7676b4f7b4-l58pk_e9cbc0c5-50e7-4d65-b53c-b5d6f0e6d75a/manager/0.log" Jul 23 18:54:24.598084 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.598050 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678z247t_69a96141-813c-41c7-bac8-54834b9fcddb/istio-proxy/0.log" Jul 23 18:54:24.823786 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.823682 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94br4_c25c8216-25cc-4161-aca2-9f98a47ecec8/discovery/0.log" Jul 23 18:54:24.882205 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.882162 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/0.log" Jul 23 18:54:24.934828 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.934792 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqpng_6a9e308b-9905-4033-8968-0edb50646e2a/istio-proxy/1.log" Jul 23 18:54:24.995162 ip-10-0-132-197 kubenswrapper[2570]: I0723 18:54:24.995131 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-c29g8_d66b6946-2d91-426e-8466-24f5075a9d8d/payload-processing/0.log"