Jul 17 05:39:11.427928 ip-10-0-136-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 05:39:11.427938 ip-10-0-136-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 05:39:11.427945 ip-10-0-136-88 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 05:39:11.428148 ip-10-0-136-88 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 05:39:21.447628 ip-10-0-136-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 05:39:21.447647 ip-10-0-136-88 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0f6e240632524c7f90135399802ce118 -- Jul 17 05:41:12.699544 ip-10-0-136-88 systemd[1]: Starting Kubernetes Kubelet... Jul 17 05:41:13.088398 ip-10-0-136-88 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:13.088398 ip-10-0-136-88 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 05:41:13.088398 ip-10-0-136-88 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:13.088398 ip-10-0-136-88 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 05:41:13.088398 ip-10-0-136-88 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 05:41:13.089207 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.089119 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 05:41:13.092974 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092958 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.092974 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092975 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092979 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092981 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092984 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092987 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092990 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092993 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092995 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.092998 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093001 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093005 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093007 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093010 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093014 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093018 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093021 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093024 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093027 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093029 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093032 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.093043 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093034 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093038 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093044 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093047 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093050 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093053 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093056 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093059 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093061 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093064 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093067 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093069 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093073 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093075 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093078 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093081 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093083 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093085 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093088 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093091 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.093579 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093093 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093095 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093098 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093100 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093102 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093105 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093107 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093109 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093111 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093115 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093119 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093122 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093126 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093128 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093131 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093134 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093136 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093139 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093141 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093144 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.094102 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093146 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093148 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093151 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093153 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093155 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093158 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093160 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093164 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093166 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093169 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093171 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093173 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093176 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093179 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093181 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093184 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093186 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093190 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093193 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.094650 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093196 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093199 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093201 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093204 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093207 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093210 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093213 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093216 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093219 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093221 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093224 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093227 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093229 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093232 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093234 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093237 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093239 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093241 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093244 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.095141 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093975 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093982 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093985 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093988 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093991 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093995 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.093999 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094003 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094006 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094010 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094012 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094015 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094017 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094020 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094022 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094025 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094028 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094030 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094033 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094035 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.095640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094038 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094041 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094044 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094046 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094049 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094051 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094054 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094056 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094059 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094062 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094064 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094067 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094069 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094071 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094073 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094076 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094078 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094081 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094084 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094086 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.096219 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094089 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094091 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094093 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094096 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094098 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094101 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094103 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094106 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094108 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094110 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094113 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094116 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094119 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094121 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094124 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094126 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094128 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094131 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094133 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094136 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.096872 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094138 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094141 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094143 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094145 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094148 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094150 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094152 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094155 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094157 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094160 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094162 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094166 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094169 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094172 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094174 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094176 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094179 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094181 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094184 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.097418 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094186 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094188 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094190 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094196 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094198 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094202 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094205 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094208 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094211 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094214 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094217 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094220 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094222 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094225 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094228 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094231 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094234 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094238 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094241 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.097913 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094244 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094624 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094628 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094631 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094634 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094637 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094640 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094643 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094646 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094648 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094651 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094653 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094656 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094659 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094661 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094664 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094668 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094671 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094675 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.098416 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094678 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094682 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094686 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094688 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094691 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094694 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094697 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094700 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094703 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094705 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094709 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094711 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094714 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094717 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094720 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094722 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094725 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094727 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094730 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094732 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.098909 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094735 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094737 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094740 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094742 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094745 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094747 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094750 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094752 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094755 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094758 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094761 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094764 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094767 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094769 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094771 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094774 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094776 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094779 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094781 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.099454 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094784 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094786 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094788 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094793 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094796 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094799 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094802 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094805 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094807 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094810 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094813 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094815 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094818 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094821 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094823 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094826 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094828 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094830 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094833 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094835 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.099965 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094837 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094840 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094844 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094847 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094849 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094852 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094854 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094857 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094859 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094862 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094864 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094867 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094869 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094872 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094874 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094877 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094879 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094882 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094884 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094886 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.100502 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094889 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.094891 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.094976 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.094984 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.094993 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.094999 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095006 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095011 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095015 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095022 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095025 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095028 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095032 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095035 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095040 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095043 2574 flags.go:64] FLAG: --cgroup-root="" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095046 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095049 2574 flags.go:64] FLAG: --client-ca-file="" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095051 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095054 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095058 2574 flags.go:64] FLAG: --cluster-domain="" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095061 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095064 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095067 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 05:41:13.101008 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095070 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095074 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095076 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095079 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095082 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095085 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095088 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095091 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095094 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095097 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095101 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095104 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095107 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095109 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095112 2574 flags.go:64] FLAG: --enable-server="true" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095117 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095122 2574 flags.go:64] FLAG: --event-burst="100" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095125 2574 flags.go:64] FLAG: --event-qps="50" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095128 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095131 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095134 2574 flags.go:64] FLAG: --eviction-hard="" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095138 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095142 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095145 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095148 2574 flags.go:64] FLAG: --eviction-soft="" Jul 17 05:41:13.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095151 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095154 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095156 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095159 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095162 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095165 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095168 2574 flags.go:64] FLAG: --feature-gates="" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095171 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095174 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095177 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095180 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095183 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095186 2574 flags.go:64] FLAG: --help="false" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095188 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095191 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095194 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095196 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095200 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095203 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095206 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095208 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095211 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095215 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 05:41:13.102869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095218 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095221 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095224 2574 flags.go:64] FLAG: --kube-reserved="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095227 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095230 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095232 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095236 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095239 2574 flags.go:64] FLAG: --lock-file="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095242 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095258 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095261 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095266 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095269 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095272 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095274 2574 flags.go:64] FLAG: --logging-format="text" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095277 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095281 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095283 2574 flags.go:64] FLAG: --manifest-url="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095286 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095290 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095293 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095297 2574 flags.go:64] FLAG: --max-pods="110" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095299 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095303 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095306 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 05:41:13.103968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095308 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095311 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095314 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095317 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095324 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095327 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095331 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095334 2574 flags.go:64] FLAG: --pod-cidr="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095337 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095342 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095345 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095348 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095351 2574 flags.go:64] FLAG: --port="10250" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095354 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095358 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-045cd68a156f00898" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095360 2574 flags.go:64] FLAG: --qos-reserved="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095363 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095366 2574 flags.go:64] FLAG: --register-node="true" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095369 2574 flags.go:64] FLAG: --register-with-taints="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095372 2574 flags.go:64] FLAG: --registry-burst="10" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095375 2574 flags.go:64] FLAG: --registry-qps="5" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095377 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095380 2574 flags.go:64] FLAG: --reserved-memory="" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095384 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095386 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 05:41:13.104897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095389 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095392 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095394 2574 flags.go:64] FLAG: --runonce="false" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095397 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095400 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095402 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095405 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095408 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095411 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095414 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095416 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095419 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095422 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095426 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095429 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095432 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095435 2574 flags.go:64] FLAG: --system-cgroups="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095437 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095442 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095445 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095447 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095453 2574 flags.go:64] FLAG: --tls-min-version="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095456 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095458 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095461 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 05:41:13.105822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095464 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095467 2574 flags.go:64] FLAG: --v="2" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095471 2574 flags.go:64] FLAG: --version="false" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095474 2574 flags.go:64] FLAG: --vmodule="" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095478 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.095481 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095593 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095597 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095600 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095603 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095606 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095609 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095611 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095614 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095616 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095619 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095621 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095624 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095626 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095629 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095633 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.106988 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095635 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095638 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095641 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095644 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095647 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095649 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095652 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095655 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095658 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095660 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095663 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095665 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095668 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095670 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095672 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095675 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095677 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095680 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095682 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.107754 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095684 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095687 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095689 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095691 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095694 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095696 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095699 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095701 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095704 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095706 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095708 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095712 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095716 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095719 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095721 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095724 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095726 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095730 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095732 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.108626 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095734 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095737 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095740 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095742 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095745 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095748 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095751 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095753 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095756 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095758 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095760 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095763 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095765 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095768 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095770 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095772 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095775 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095778 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095780 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095782 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.109287 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095785 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095787 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095790 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095792 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095794 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095797 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095800 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095802 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095805 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095807 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095810 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095812 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095815 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095817 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095821 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095824 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095827 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095829 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095832 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.109945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095834 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095838 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095841 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095843 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095846 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095849 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.095852 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.096432 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.104506 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.104529 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.105298 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.105314 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105419 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105427 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.110488 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105433 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105438 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105442 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105446 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105450 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105455 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105459 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105463 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105467 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105471 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105475 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105479 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105484 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105488 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105492 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105497 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105501 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105505 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105509 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105513 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.110864 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105517 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105521 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105526 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105530 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105534 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105538 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105542 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105558 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105566 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105570 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105574 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105579 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105583 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105587 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105591 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105595 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105599 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105603 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105608 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.111467 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105613 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105617 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105621 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105625 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105629 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105633 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105637 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105641 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105645 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105649 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105653 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105657 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105661 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105665 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105669 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105676 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105682 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105687 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105691 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105696 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.112059 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105701 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105713 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105717 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105721 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105725 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105729 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105733 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105738 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105742 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105746 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105751 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105755 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105759 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105763 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105767 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105771 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105775 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105779 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105785 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.112792 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105791 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105796 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105800 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105805 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105809 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105814 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105818 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105822 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105827 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105832 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105836 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105839 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105843 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105847 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105852 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105864 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105869 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105873 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.113305 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.105877 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.105886 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106109 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106122 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106129 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106134 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106139 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106144 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106149 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106153 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106157 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106161 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106165 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106169 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106173 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 05:41:13.114006 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106177 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106182 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106186 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106190 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106194 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106198 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106204 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106210 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106217 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106223 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106228 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106233 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106239 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106244 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106269 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106282 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106287 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106291 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106295 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 05:41:13.114619 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106299 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106304 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106308 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106312 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106317 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106321 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106325 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106329 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106333 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106337 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106341 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106346 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106350 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106354 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106358 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106362 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106366 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106370 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106374 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106378 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 05:41:13.115233 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106382 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106386 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106391 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106395 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106399 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106403 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106407 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106411 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106415 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106430 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106435 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106439 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106443 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106447 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106451 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106455 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106459 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106463 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106468 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106472 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 05:41:13.115903 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106476 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106479 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106483 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106488 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106492 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106496 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106500 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106504 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106508 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106512 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106516 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106520 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106524 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106529 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106533 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106537 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106541 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106545 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106549 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106553 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 05:41:13.116522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106557 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106561 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106573 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106578 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106582 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106586 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:13.106590 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.106599 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.106778 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.109610 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.110965 2574 server.go:1019] "Starting client certificate rotation" Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.111082 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.111118 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 05:41:13.117038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.116301 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 05:41:13.117809 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.117792 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 05:41:13.132820 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.132797 2574 log.go:25] "Validated CRI v1 runtime API" Jul 17 05:41:13.134747 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.134733 2574 log.go:25] "Validated CRI v1 image API" Jul 17 05:41:13.136530 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.136514 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 05:41:13.139492 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.139474 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 93b886bc-b503-4880-9b61-466d23425860:/dev/nvme0n1p4 c02503cc-73ca-48e5-832a-d4110d49cc0e:/dev/nvme0n1p3] Jul 17 05:41:13.139541 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.139493 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 05:41:13.143410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.143386 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 05:41:13.145416 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.145305 2574 manager.go:216] Machine: {Timestamp:2026-07-17 05:41:13.143450327 +0000 UTC m=+0.332830605 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099304 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f2e4739b799d8c43ca7087b83d26 SystemUUID:ec21f2e4-739b-799d-8c43-ca7087b83d26 BootID:0f6e2406-3252-4c7f-9013-5399802ce118 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:d9:77:54:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:d9:77:54:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:27:1f:2e:fa:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 05:41:13.145416 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.145411 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 05:41:13.145534 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.145494 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 05:41:13.148440 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.148410 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 05:41:13.148573 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.148444 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 05:41:13.149097 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.149087 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 05:41:13.149129 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.149098 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 05:41:13.149129 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.149105 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 05:41:13.149734 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.149723 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 05:41:13.150866 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.150857 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 05:41:13.153872 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.153860 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 17 05:41:13.154000 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.153991 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 05:41:13.154040 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154031 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 17 05:41:13.154076 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154047 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 05:41:13.154125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154116 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 05:41:13.154156 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154133 2574 kubelet.go:398] "Adding apiserver pod source" Jul 17 05:41:13.154156 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154144 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 05:41:13.154802 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154784 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 05:41:13.154802 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.154813 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 05:41:13.157814 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.157795 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 05:41:13.159657 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.159643 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 05:41:13.159724 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.159673 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 05:41:13.160657 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160646 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 05:41:13.160696 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160677 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 05:41:13.160696 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160685 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 05:41:13.160696 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160690 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160699 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160705 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160713 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160719 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160725 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160731 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160739 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 05:41:13.160803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.160744 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 05:41:13.162329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.162318 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 05:41:13.162329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.162329 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 05:41:13.164473 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.164448 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 05:41:13.164546 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.164487 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 05:41:13.164546 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.164502 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 05:41:13.165650 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.165638 2574 server.go:1268] "Started kubelet" Jul 17 05:41:13.165721 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.165697 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 05:41:13.165814 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.165727 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 05:41:13.165859 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.165841 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 05:41:13.166515 ip-10-0-136-88 systemd[1]: Started Kubernetes Kubelet. Jul 17 05:41:13.167439 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.167340 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 05:41:13.168117 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.168097 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 17 05:41:13.170617 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.169806 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c2fd4487887d60 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-17 05:41:13.165610336 +0000 UTC m=+0.354990610,LastTimestamp:2026-07-17 05:41:13.165610336 +0000 UTC m=+0.354990610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}" Jul 17 05:41:13.171222 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.171206 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 05:41:13.173854 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.173835 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 05:41:13.173951 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.173843 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 05:41:13.174005 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.173950 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.174548 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174530 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 05:41:13.174641 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174579 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 05:41:13.174641 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174591 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 05:41:13.174758 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174735 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 17 05:41:13.174758 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174752 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 17 05:41:13.174845 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174784 2574 factory.go:55] Registering systemd factory Jul 17 05:41:13.174845 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.174798 2574 factory.go:223] Registration of the systemd container factory successfully Jul 17 05:41:13.175202 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175188 2574 factory.go:153] Registering CRI-O factory Jul 17 05:41:13.175282 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175206 2574 factory.go:223] Registration of the crio container factory successfully Jul 17 05:41:13.175317 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175285 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 05:41:13.175359 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175322 2574 factory.go:103] Registering Raw factory Jul 17 05:41:13.175359 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175348 2574 manager.go:1196] Started watching for new ooms in manager Jul 17 05:41:13.176009 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.175993 2574 manager.go:349] Starting recovery of all containers Jul 17 05:41:13.185055 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.185024 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 05:41:13.185185 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.185097 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 05:41:13.189014 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.188853 2574 manager.go:354] Recovery completed Jul 17 05:41:13.192075 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.192052 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6xtz" Jul 17 05:41:13.193583 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.193571 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.195465 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.195450 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.195525 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.195481 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.195525 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.195494 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.196047 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196032 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 05:41:13.196047 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196044 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 05:41:13.196125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196046 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.196125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196063 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 17 05:41:13.196784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196770 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.196846 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196796 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.196846 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.196806 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.197586 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.197574 2574 policy_none.go:49] "None policy: Start" Jul 17 05:41:13.197629 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.197591 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 05:41:13.197629 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.197603 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 05:41:13.198693 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.198683 2574 policy_none.go:47] "Start" Jul 17 05:41:13.199600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.199585 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6xtz" Jul 17 05:41:13.229796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.229773 2574 manager.go:337] "Starting Device Plugin manager" Jul 17 05:41:13.229904 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.229811 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 05:41:13.229904 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.229825 2574 server.go:87] "Starting device plugin registration server" Jul 17 05:41:13.230084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.230069 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 05:41:13.230141 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.230088 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 05:41:13.230211 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.230194 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 05:41:13.230342 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.230291 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 05:41:13.230342 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.230304 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 05:41:13.230857 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.230839 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 05:41:13.230934 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.230882 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.274201 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.274160 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 05:41:13.275633 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.275604 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 05:41:13.275713 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.275650 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 05:41:13.275713 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.275669 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 05:41:13.275713 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.275700 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 05:41:13.278312 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.278292 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:13.331087 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.331059 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.332186 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.332168 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.332266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.332201 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.332266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.332213 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.332266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.332243 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.341976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.341929 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.341976 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.341956 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-88.ec2.internal\": node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.367081 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.367064 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.376310 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.376291 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"] Jul 17 05:41:13.376361 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.376355 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.377188 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.377159 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.377188 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.377187 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.377332 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.377198 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.378457 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.378445 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.378559 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.378545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.378600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.378578 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.379136 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379118 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.379210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379148 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.379210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379162 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.379210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379165 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.379210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379187 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.379210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.379202 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.380310 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.380297 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.380347 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.380320 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 05:41:13.380959 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.380938 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory" Jul 17 05:41:13.381022 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.380966 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 05:41:13.381022 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.380977 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID" Jul 17 05:41:13.402784 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.402765 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-88.ec2.internal\" not found" node="ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.406848 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.406832 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-88.ec2.internal\" not found" node="ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.467939 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.467920 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.477317 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.477295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.477384 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.477318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.477384 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.477335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3544f86c6a812cb27ca33d092eba6944-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"3544f86c6a812cb27ca33d092eba6944\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.568445 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.568421 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.577771 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.577831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3544f86c6a812cb27ca33d092eba6944-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"3544f86c6a812cb27ca33d092eba6944\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.577831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.577831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.577923 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d90eecfe8dc670a9d64268d2eefb859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"4d90eecfe8dc670a9d64268d2eefb859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.577923 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.577853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3544f86c6a812cb27ca33d092eba6944-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"3544f86c6a812cb27ca33d092eba6944\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.669273 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.669180 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.705734 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.705714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.710118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.709895 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:13.769723 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.769696 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.870203 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.870175 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.970813 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:13.970750 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:13.980266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:13.980231 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:14.071199 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.071170 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.110803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.110768 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 05:41:14.111329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.110939 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 05:41:14.111329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.110955 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 05:41:14.171725 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.171696 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.204217 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.204180 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 05:36:14 +0000 UTC" deadline="2028-02-16 01:45:41.376449589 +0000 UTC" Jul 17 05:41:14.204377 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.204215 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13892h4m27.172238223s" Jul 17 05:41:14.272545 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.272478 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.287517 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:14.287484 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d90eecfe8dc670a9d64268d2eefb859.slice/crio-93d26569fd42c3b1d76b4af9285bbdb09024613eb6c899c109fb04eae7dcbf8c WatchSource:0}: Error finding container 93d26569fd42c3b1d76b4af9285bbdb09024613eb6c899c109fb04eae7dcbf8c: Status 404 returned error can't find the container with id 93d26569fd42c3b1d76b4af9285bbdb09024613eb6c899c109fb04eae7dcbf8c Jul 17 05:41:14.287797 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:14.287782 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3544f86c6a812cb27ca33d092eba6944.slice/crio-d22eca59c0f3355292e69b1ac95ea3b8526882847ddc71a3c34625d77eed9a52 WatchSource:0}: Error finding container d22eca59c0f3355292e69b1ac95ea3b8526882847ddc71a3c34625d77eed9a52: Status 404 returned error can't find the container with id d22eca59c0f3355292e69b1ac95ea3b8526882847ddc71a3c34625d77eed9a52 Jul 17 05:41:14.293997 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.293981 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:41:14.374196 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.374165 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.423506 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.423480 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:14.476988 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.476960 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.577526 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.577447 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.677735 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:14.677700 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found" Jul 17 05:41:14.716656 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.716633 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:14.777778 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.777742 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" Jul 17 05:41:14.790711 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.790689 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 05:41:14.791606 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.791587 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" Jul 17 05:41:14.800358 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:14.800343 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 05:41:15.145969 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.145934 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:15.155382 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.155134 2574 apiserver.go:52] "Watching apiserver" Jul 17 05:41:15.165022 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.164998 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 05:41:15.165530 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.165502 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-lswjg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn","openshift-network-diagnostics/network-check-target-s4lkz","openshift-network-operator/iptables-alerter-lvdw6","openshift-ovn-kubernetes/ovnkube-node-lgqkw","kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vs4v5","openshift-dns/node-resolver-dnm5g","openshift-image-registry/node-ca-pft2v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal","openshift-multus/multus-additional-cni-plugins-z9flj","openshift-multus/multus-r4lfx","openshift-multus/network-metrics-daemon-gr7tv","kube-system/global-pull-secret-syncer-g7rcl"] Jul 17 05:41:15.167354 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.167331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.169091 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.169067 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.169193 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.169158 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:15.170109 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.170084 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.170843 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.170288 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.170843 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.170519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ht6v2\"" Jul 17 05:41:15.170843 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.170712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:15.170843 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.170769 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:15.173465 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.172442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.173465 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.173251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.175088 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.174431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.175088 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.174506 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:15.175595 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.175540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.175819 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.175798 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 05:41:15.176021 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.176003 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hl865\"" Jul 17 05:41:15.177674 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.177652 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 05:41:15.178340 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.178302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 05:41:15.178637 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.178610 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 05:41:15.178738 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.178716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 05:41:15.178810 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.178735 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.178871 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.177797 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 05:41:15.180205 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.179207 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gfxfr\"" Jul 17 05:41:15.180205 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.177875 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.180205 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.179812 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.180996 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.180977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.181860 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.181827 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.181958 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.181933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.182181 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.182160 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.182730 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.182708 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 05:41:15.182900 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.182882 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.183633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mf65m\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.183966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.186173 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.186525 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lxcrg\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.186793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187041 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187380 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187416 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187680 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187415 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-52s9f\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-dbus\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-netns\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-log-socket\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-env-overrides\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-tuned\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.187953 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wwp7t\"" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-konnectivity-ca\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-run\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsjd\" (UniqueName: \"kubernetes.io/projected/9dd9f06f-83b5-455a-87c0-0942e531e66c-kube-api-access-gnsjd\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-lib-modules\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.189706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-host\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-tmp\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvbq\" (UniqueName: \"kubernetes.io/projected/7dbdb628-ab21-484c-9b77-17e81ddd6698-kube-api-access-nfvbq\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysconfig\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-netd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-sys\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-socket-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-kubernetes\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-systemd\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-systemd-units\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-device-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-var-lib-kubelet\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-kubelet\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.192380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-slash\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aafc48fa-52d1-44c9-9b9d-004664557555-ovn-node-metrics-cert\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-script-lib\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxh7\" (UniqueName: \"kubernetes.io/projected/aafc48fa-52d1-44c9-9b9d-004664557555-kube-api-access-2jxh7\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-etc-selinux\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f92364b-d158-40b9-927a-b60fa2610dd1-host-slash\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-agent-certs\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgj6\" (UniqueName: \"kubernetes.io/projected/7f92364b-d158-40b9-927a-b60fa2610dd1-kube-api-access-qfgj6\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-modprobe-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.188983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-bin\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-conf\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2f8d\" (UniqueName: \"kubernetes.io/projected/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-kube-api-access-f2f8d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-var-lib-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-config\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193041 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-etc-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-ovn\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-node-log\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dbdb628-ab21-484c-9b77-17e81ddd6698-host\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbdb628-ab21-484c-9b77-17e81ddd6698-serviceca\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-sys-fs\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-kubelet-config\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-registration-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9qq\" (UniqueName: \"kubernetes.io/projected/6617226d-ebd9-40ab-ae75-4fec0d433e35-kube-api-access-dl9qq\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f92364b-d158-40b9-927a-b60fa2610dd1-iptables-alerter-script\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.189726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-systemd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.190733 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.191114 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.192268 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.192368 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.192790 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.193695 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 05:41:15.193737 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.193739 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 05:41:15.194755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.194043 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 05:41:15.194755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.194059 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 05:41:15.194755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.194259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjr4m\"" Jul 17 05:41:15.194755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.194559 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 05:41:15.194755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.194589 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-66c5t\"" Jul 17 05:41:15.217836 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.217802 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8lnk9" Jul 17 05:41:15.222652 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.222607 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8lnk9" Jul 17 05:41:15.275966 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.275940 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 05:41:15.290230 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-konnectivity-ca\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-run\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-os-release\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-k8s-cni-cncf-io\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsjd\" (UniqueName: \"kubernetes.io/projected/9dd9f06f-83b5-455a-87c0-0942e531e66c-kube-api-access-gnsjd\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-lib-modules\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-run\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-host\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-tmp\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-multus\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvbq\" (UniqueName: \"kubernetes.io/projected/7dbdb628-ab21-484c-9b77-17e81ddd6698-kube-api-access-nfvbq\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-cnibin\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysconfig\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-netd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-socket-dir-parent\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-sys\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-konnectivity-ca\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.290939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-socket-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysconfig\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-kubernetes\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.290998 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-systemd\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-systemd-units\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-lib-modules\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cnibin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-host\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-device-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-systemd-units\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291261 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-kubernetes\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.291582 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-netd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-sys\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-var-lib-kubelet\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-kubelet\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-slash\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aafc48fa-52d1-44c9-9b9d-004664557555-ovn-node-metrics-cert\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-script-lib\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxh7\" (UniqueName: \"kubernetes.io/projected/aafc48fa-52d1-44c9-9b9d-004664557555-kube-api-access-2jxh7\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.291910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-os-release\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-socket-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-etc-selinux\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f92364b-d158-40b9-927a-b60fa2610dd1-host-slash\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnq5d\" (UniqueName: \"kubernetes.io/projected/cf333d5d-2436-4afe-910c-e79c9cd7fa72-kube-api-access-pnq5d\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhvh\" (UniqueName: \"kubernetes.io/projected/3df423e1-e9ed-45bc-ac94-f443e1686291-kube-api-access-vdhvh\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-hostroot\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292291 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgbh\" (UniqueName: \"kubernetes.io/projected/477a7d2a-8f1a-47ea-9956-0d49528c80f0-kube-api-access-mlgbh\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.292367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-agent-certs\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgj6\" (UniqueName: \"kubernetes.io/projected/7f92364b-d158-40b9-927a-b60fa2610dd1-kube-api-access-qfgj6\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-etc-selinux\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-systemd\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-system-cni-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-modprobe-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-bin\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-netns\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-etc-kubernetes\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-script-lib\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292884 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f92364b-d158-40b9-927a-b60fa2610dd1-host-slash\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.292789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-conf\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-conf\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-var-lib-kubelet\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293161 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-kubelet\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-cni-bin\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-modprobe-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-slash\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-multus-certs\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-device-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2f8d\" (UniqueName: \"kubernetes.io/projected/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-kube-api-access-f2f8d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-var-lib-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-config\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf333d5d-2436-4afe-910c-e79c9cd7fa72-tmp-dir\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-sysctl-d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-conf-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-etc-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-ovn\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.293907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-node-log\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.293913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dbdb628-ab21-484c-9b77-17e81ddd6698-host\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbdb628-ab21-484c-9b77-17e81ddd6698-serviceca\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf333d5d-2436-4afe-910c-e79c9cd7fa72-hosts-file\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-sys-fs\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-kubelet-config\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dbdb628-ab21-484c-9b77-17e81ddd6698-host\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cni-binary-copy\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-bin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-kubelet\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-daemon-config\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.294896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-registration-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.295602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9qq\" (UniqueName: \"kubernetes.io/projected/6617226d-ebd9-40ab-ae75-4fec0d433e35-kube-api-access-dl9qq\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.295602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.294972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f92364b-d158-40b9-927a-b60fa2610dd1-iptables-alerter-script\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.295602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-systemd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.295602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-etc-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.295602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-ovn\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.295853 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.295622 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:15.295853 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-dbus\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.295853 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-run-systemd\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.295853 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-system-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.295853 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.295708 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:15.795686141 +0000 UTC m=+2.985066426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:15.296121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-var-lib-openvswitch\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.295980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-netns\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.296121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-log-socket\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-env-overrides\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dbdb628-ab21-484c-9b77-17e81ddd6698-serviceca\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-tuned\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-ovnkube-config\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-sys-fs\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-log-socket\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.296494 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-dbus\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.296915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7f92364b-d158-40b9-927a-b60fa2610dd1-iptables-alerter-script\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.296915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aafc48fa-52d1-44c9-9b9d-004664557555-env-overrides\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.297058 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-kubelet-config\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.297058 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.296999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-run-netns\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.297167 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.297074 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:15.297167 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.297126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9dd9f06f-83b5-455a-87c0-0942e531e66c-registration-dir\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.298344 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.297866 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:15.298344 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.297895 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:15.298344 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.297911 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:15.300112 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.300055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aafc48fa-52d1-44c9-9b9d-004664557555-ovn-node-metrics-cert\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.300112 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.300123 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54f7b06a-0ab5-4306-8c29-c0e86a4762cf-agent-certs\") pod \"konnectivity-agent-lswjg\" (UID: \"54f7b06a-0ab5-4306-8c29-c0e86a4762cf\") " pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.300453 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.300417 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-tmp\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.302252 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.302187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-etc-tuned\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.302471 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.302326 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:15.802304308 +0000 UTC m=+2.991684589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:15.302583 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.302566 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:15.802547454 +0000 UTC m=+2.991927736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:15.303575 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.303018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsjd\" (UniqueName: \"kubernetes.io/projected/9dd9f06f-83b5-455a-87c0-0942e531e66c-kube-api-access-gnsjd\") pod \"aws-ebs-csi-driver-node-l82vn\" (UID: \"9dd9f06f-83b5-455a-87c0-0942e531e66c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.303575 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.303406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvbq\" (UniqueName: \"kubernetes.io/projected/7dbdb628-ab21-484c-9b77-17e81ddd6698-kube-api-access-nfvbq\") pod \"node-ca-pft2v\" (UID: \"7dbdb628-ab21-484c-9b77-17e81ddd6698\") " pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.304219 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.304052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.304534 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.304511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgj6\" (UniqueName: \"kubernetes.io/projected/7f92364b-d158-40b9-927a-b60fa2610dd1-kube-api-access-qfgj6\") pod \"iptables-alerter-lvdw6\" (UID: \"7f92364b-d158-40b9-927a-b60fa2610dd1\") " pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.304628 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.304522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aafc48fa-52d1-44c9-9b9d-004664557555-node-log\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.305606 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.305552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" event={"ID":"3544f86c6a812cb27ca33d092eba6944","Type":"ContainerStarted","Data":"d22eca59c0f3355292e69b1ac95ea3b8526882847ddc71a3c34625d77eed9a52"} Jul 17 05:41:15.305707 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.305631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"4d90eecfe8dc670a9d64268d2eefb859","Type":"ContainerStarted","Data":"93d26569fd42c3b1d76b4af9285bbdb09024613eb6c899c109fb04eae7dcbf8c"} Jul 17 05:41:15.305865 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.305839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxh7\" (UniqueName: \"kubernetes.io/projected/aafc48fa-52d1-44c9-9b9d-004664557555-kube-api-access-2jxh7\") pod \"ovnkube-node-lgqkw\" (UID: \"aafc48fa-52d1-44c9-9b9d-004664557555\") " pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.308778 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.308752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2f8d\" (UniqueName: \"kubernetes.io/projected/b18fc67e-4aff-40f1-a2cb-ad954e2829eb-kube-api-access-f2f8d\") pod \"tuned-vs4v5\" (UID: \"b18fc67e-4aff-40f1-a2cb-ad954e2829eb\") " pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.311769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.311742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9qq\" (UniqueName: \"kubernetes.io/projected/6617226d-ebd9-40ab-ae75-4fec0d433e35-kube-api-access-dl9qq\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.396974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.396891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cnibin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.396974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.396936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-os-release\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.396974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.396957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnq5d\" (UniqueName: \"kubernetes.io/projected/cf333d5d-2436-4afe-910c-e79c9cd7fa72-kube-api-access-pnq5d\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.396983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhvh\" (UniqueName: \"kubernetes.io/projected/3df423e1-e9ed-45bc-ac94-f443e1686291-kube-api-access-vdhvh\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-hostroot\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cnibin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-hostroot\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-os-release\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgbh\" (UniqueName: \"kubernetes.io/projected/477a7d2a-8f1a-47ea-9956-0d49528c80f0-kube-api-access-mlgbh\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-system-cni-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.397234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-netns\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-system-cni-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-netns\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-etc-kubernetes\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-multus-certs\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-etc-kubernetes\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf333d5d-2436-4afe-910c-e79c9cd7fa72-tmp-dir\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-multus-certs\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-conf-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf333d5d-2436-4afe-910c-e79c9cd7fa72-hosts-file\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-conf-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cni-binary-copy\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397540 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf333d5d-2436-4afe-910c-e79c9cd7fa72-hosts-file\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-bin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-kubelet\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.397634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-daemon-config\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-system-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf333d5d-2436-4afe-910c-e79c9cd7fa72-tmp-dir\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-bin\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397705 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-system-cni-dir\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-os-release\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-kubelet\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-k8s-cni-cncf-io\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-multus\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-os-release\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-cnibin\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-run-k8s-cni-cncf-io\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-socket-dir-parent\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-host-var-lib-cni-multus\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398245 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.397945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-daemon-config\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/477a7d2a-8f1a-47ea-9956-0d49528c80f0-cni-binary-copy\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/477a7d2a-8f1a-47ea-9956-0d49528c80f0-multus-socket-dir-parent\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df423e1-e9ed-45bc-ac94-f443e1686291-cnibin\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.398920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.398420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.399210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.399086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3df423e1-e9ed-45bc-ac94-f443e1686291-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.405513 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.405488 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhvh\" (UniqueName: \"kubernetes.io/projected/3df423e1-e9ed-45bc-ac94-f443e1686291-kube-api-access-vdhvh\") pod \"multus-additional-cni-plugins-z9flj\" (UID: \"3df423e1-e9ed-45bc-ac94-f443e1686291\") " pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.405802 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.405780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnq5d\" (UniqueName: \"kubernetes.io/projected/cf333d5d-2436-4afe-910c-e79c9cd7fa72-kube-api-access-pnq5d\") pod \"node-resolver-dnm5g\" (UID: \"cf333d5d-2436-4afe-910c-e79c9cd7fa72\") " pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.405856 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.405792 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgbh\" (UniqueName: \"kubernetes.io/projected/477a7d2a-8f1a-47ea-9956-0d49528c80f0-kube-api-access-mlgbh\") pod \"multus-r4lfx\" (UID: \"477a7d2a-8f1a-47ea-9956-0d49528c80f0\") " pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.494113 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.494082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" Jul 17 05:41:15.504997 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.504973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvdw6" Jul 17 05:41:15.505352 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:15.505306 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18fc67e_4aff_40f1_a2cb_ad954e2829eb.slice/crio-8da51f10edf502aa57750dae265f3a53bfd3953bc97a728a2e0209dc36f69e72 WatchSource:0}: Error finding container 8da51f10edf502aa57750dae265f3a53bfd3953bc97a728a2e0209dc36f69e72: Status 404 returned error can't find the container with id 8da51f10edf502aa57750dae265f3a53bfd3953bc97a728a2e0209dc36f69e72 Jul 17 05:41:15.513944 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.513923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:15.514163 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:15.514136 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f92364b_d158_40b9_927a_b60fa2610dd1.slice/crio-c7c00118554f30179ba6b2894713b636e72f594ad5ee03421d4f2c941cc4baa2 WatchSource:0}: Error finding container c7c00118554f30179ba6b2894713b636e72f594ad5ee03421d4f2c941cc4baa2: Status 404 returned error can't find the container with id c7c00118554f30179ba6b2894713b636e72f594ad5ee03421d4f2c941cc4baa2 Jul 17 05:41:15.522493 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.522464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pft2v" Jul 17 05:41:15.524369 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:15.524203 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafc48fa_52d1_44c9_9b9d_004664557555.slice/crio-b10a21b477a25478475245daa53be0a1bf5c5da282881e474d5ce55ebee1b110 WatchSource:0}: Error finding container b10a21b477a25478475245daa53be0a1bf5c5da282881e474d5ce55ebee1b110: Status 404 returned error can't find the container with id b10a21b477a25478475245daa53be0a1bf5c5da282881e474d5ce55ebee1b110 Jul 17 05:41:15.529649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.529628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:15.537515 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.537489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" Jul 17 05:41:15.546408 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.546388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dnm5g" Jul 17 05:41:15.556187 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.556166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9flj" Jul 17 05:41:15.566197 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.566175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4lfx" Jul 17 05:41:15.801711 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.801564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:15.801711 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.801706 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:15.801947 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.801781 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:16.801758367 +0000 UTC m=+3.991138634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:15.902357 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.902329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:15.902483 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:15.902370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:15.902527 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902480 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:15.902527 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902493 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:15.902527 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902506 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:15.902527 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902515 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:15.902680 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902540 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:16.902525104 +0000 UTC m=+4.091905371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:15.902680 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:15.902559 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:16.902547534 +0000 UTC m=+4.091927796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:16.225553 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.225465 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 05:36:16 +0000 UTC" deadline="2028-04-04 10:25:09.003927504 +0000 UTC" Jul 17 05:41:16.225553 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.225498 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15052h43m52.778434388s" Jul 17 05:41:16.234267 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.234224 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbdb628_ab21_484c_9b77_17e81ddd6698.slice/crio-c8b64377451f6956a54f14e7288a1e0d08848b6d5d129a75df7b69ff89ea397f WatchSource:0}: Error finding container c8b64377451f6956a54f14e7288a1e0d08848b6d5d129a75df7b69ff89ea397f: Status 404 returned error can't find the container with id c8b64377451f6956a54f14e7288a1e0d08848b6d5d129a75df7b69ff89ea397f Jul 17 05:41:16.287687 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.287648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvdw6" event={"ID":"7f92364b-d158-40b9-927a-b60fa2610dd1","Type":"ContainerStarted","Data":"c7c00118554f30179ba6b2894713b636e72f594ad5ee03421d4f2c941cc4baa2"} Jul 17 05:41:16.289122 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.289093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" event={"ID":"b18fc67e-4aff-40f1-a2cb-ad954e2829eb","Type":"ContainerStarted","Data":"8da51f10edf502aa57750dae265f3a53bfd3953bc97a728a2e0209dc36f69e72"} Jul 17 05:41:16.290402 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.290376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pft2v" event={"ID":"7dbdb628-ab21-484c-9b77-17e81ddd6698","Type":"ContainerStarted","Data":"c8b64377451f6956a54f14e7288a1e0d08848b6d5d129a75df7b69ff89ea397f"} Jul 17 05:41:16.291706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.291677 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"b10a21b477a25478475245daa53be0a1bf5c5da282881e474d5ce55ebee1b110"} Jul 17 05:41:16.612053 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.612024 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df423e1_e9ed_45bc_ac94_f443e1686291.slice/crio-cfbe2b3eb7f1d71d803365e91f3ad9e4ca4f56a0e7d2fb0a0af49d1169f0c348 WatchSource:0}: Error finding container cfbe2b3eb7f1d71d803365e91f3ad9e4ca4f56a0e7d2fb0a0af49d1169f0c348: Status 404 returned error can't find the container with id cfbe2b3eb7f1d71d803365e91f3ad9e4ca4f56a0e7d2fb0a0af49d1169f0c348 Jul 17 05:41:16.612596 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.612564 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd9f06f_83b5_455a_87c0_0942e531e66c.slice/crio-f4388523189a09d17d931924fbb7238183a56905797e009a21c0ce9afc4c8e91 WatchSource:0}: Error finding container f4388523189a09d17d931924fbb7238183a56905797e009a21c0ce9afc4c8e91: Status 404 returned error can't find the container with id f4388523189a09d17d931924fbb7238183a56905797e009a21c0ce9afc4c8e91 Jul 17 05:41:16.614071 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.613971 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf333d5d_2436_4afe_910c_e79c9cd7fa72.slice/crio-61f27f1573eb099b32c70c2f998cbbdacc5982b3a9cd7b8e0ed9bf1a9469e2e5 WatchSource:0}: Error finding container 61f27f1573eb099b32c70c2f998cbbdacc5982b3a9cd7b8e0ed9bf1a9469e2e5: Status 404 returned error can't find the container with id 61f27f1573eb099b32c70c2f998cbbdacc5982b3a9cd7b8e0ed9bf1a9469e2e5 Jul 17 05:41:16.614704 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.614664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477a7d2a_8f1a_47ea_9956_0d49528c80f0.slice/crio-46fea631ca03b6a080a0430e60f5299493007ce70535f384706eba6e95137d27 WatchSource:0}: Error finding container 46fea631ca03b6a080a0430e60f5299493007ce70535f384706eba6e95137d27: Status 404 returned error can't find the container with id 46fea631ca03b6a080a0430e60f5299493007ce70535f384706eba6e95137d27 Jul 17 05:41:16.616169 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:16.616117 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f7b06a_0ab5_4306_8c29_c0e86a4762cf.slice/crio-41c45ee105618f446d04683d7e3379284fbbc4345e32c1e543ce3866c07134fd WatchSource:0}: Error finding container 41c45ee105618f446d04683d7e3379284fbbc4345e32c1e543ce3866c07134fd: Status 404 returned error can't find the container with id 41c45ee105618f446d04683d7e3379284fbbc4345e32c1e543ce3866c07134fd Jul 17 05:41:16.810504 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.810459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:16.810697 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.810674 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:16.810790 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.810747 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.810726346 +0000 UTC m=+6.000106627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:16.911341 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.911287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:16.911458 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:16.911356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:16.911562 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.911548 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:16.911627 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.911574 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:16.911627 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.911589 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:16.911738 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.911646 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.911624635 +0000 UTC m=+6.101004914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:16.912059 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.912037 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:16.912159 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:16.912093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:18.912077761 +0000 UTC m=+6.101458029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:17.144289 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.144182 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 05:41:17.226315 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.226226 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 05:36:16 +0000 UTC" deadline="2028-04-10 08:07:44.169633098 +0000 UTC" Jul 17 05:41:17.226315 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.226312 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15194h26m26.943326263s" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.279168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:17.279280 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.279700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:17.279778 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.279849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:17.286300 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:17.279917 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:17.298655 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.298445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerStarted","Data":"cfbe2b3eb7f1d71d803365e91f3ad9e4ca4f56a0e7d2fb0a0af49d1169f0c348"} Jul 17 05:41:17.302609 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.302577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" event={"ID":"3544f86c6a812cb27ca33d092eba6944","Type":"ContainerStarted","Data":"daa12ba6f0b169c27db7c962007b816290e43cb803b3ebc28a32c60bde023703"} Jul 17 05:41:17.306342 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.305565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4lfx" event={"ID":"477a7d2a-8f1a-47ea-9956-0d49528c80f0","Type":"ContainerStarted","Data":"46fea631ca03b6a080a0430e60f5299493007ce70535f384706eba6e95137d27"} Jul 17 05:41:17.312134 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.312103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dnm5g" event={"ID":"cf333d5d-2436-4afe-910c-e79c9cd7fa72","Type":"ContainerStarted","Data":"61f27f1573eb099b32c70c2f998cbbdacc5982b3a9cd7b8e0ed9bf1a9469e2e5"} Jul 17 05:41:17.321581 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.321525 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" podStartSLOduration=2.321482318 podStartE2EDuration="2.321482318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:17.320539799 +0000 UTC m=+4.509920086" watchObservedRunningTime="2026-07-17 05:41:17.321482318 +0000 UTC m=+4.510862605" Jul 17 05:41:17.325605 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.325573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" event={"ID":"9dd9f06f-83b5-455a-87c0-0942e531e66c","Type":"ContainerStarted","Data":"f4388523189a09d17d931924fbb7238183a56905797e009a21c0ce9afc4c8e91"} Jul 17 05:41:17.331139 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.331110 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d90eecfe8dc670a9d64268d2eefb859" containerID="ab00316b5982582bb32d4d5324a1fd0bc7268f53d4d0fba0688ad267a7190519" exitCode=0 Jul 17 05:41:17.331261 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.331197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"4d90eecfe8dc670a9d64268d2eefb859","Type":"ContainerDied","Data":"ab00316b5982582bb32d4d5324a1fd0bc7268f53d4d0fba0688ad267a7190519"} Jul 17 05:41:17.345854 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:17.345283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lswjg" event={"ID":"54f7b06a-0ab5-4306-8c29-c0e86a4762cf","Type":"ContainerStarted","Data":"41c45ee105618f446d04683d7e3379284fbbc4345e32c1e543ce3866c07134fd"} Jul 17 05:41:18.362606 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:18.362566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"4d90eecfe8dc670a9d64268d2eefb859","Type":"ContainerStarted","Data":"6ef6779e5239519c3f655ffee74d78d2fa3dcd48d8e711b049435c216942af5c"} Jul 17 05:41:18.380160 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:18.380097 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" podStartSLOduration=3.38007343 podStartE2EDuration="3.38007343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:18.379698955 +0000 UTC m=+5.569079244" watchObservedRunningTime="2026-07-17 05:41:18.38007343 +0000 UTC m=+5.569453718" Jul 17 05:41:18.826346 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:18.826234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:18.826528 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.826400 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.826528 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.826472 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:22.826448775 +0000 UTC m=+10.015829039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:18.927410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:18.927300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:18.927602 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:18.927417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:18.927602 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.927554 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.927725 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.927623 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:22.927600691 +0000 UTC m=+10.116980995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:18.928089 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.928056 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:18.928089 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.928086 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:18.928237 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.928099 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:18.928237 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:18.928154 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:22.928136638 +0000 UTC m=+10.117516924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:19.276687 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:19.276603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:19.276851 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:19.276764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:19.277220 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:19.277196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:19.277352 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:19.277333 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:19.277509 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:19.277484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:19.277604 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:19.277585 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:21.279412 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:21.279551 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:21.279932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:21.280028 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:21.280082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:21.280243 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:21.280170 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:22.863899 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:22.863824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:22.864462 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.863998 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:22.864462 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.864071 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:30.864049106 +0000 UTC m=+18.053429375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:22.964749 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:22.964713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:22.964945 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:22.964770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:22.964945 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.964909 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:22.964945 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.964912 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:22.964945 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.964933 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:22.964945 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.964947 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:22.965200 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.964991 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:30.964967514 +0000 UTC m=+18.154347792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:22.965200 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:22.965013 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:30.965001679 +0000 UTC m=+18.154381945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:23.282227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:23.282117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:23.282414 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:23.282279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:23.282414 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:23.282339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:23.282414 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:23.282350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:23.282570 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:23.282460 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:23.282629 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:23.282610 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:25.276385 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:25.276348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:25.276385 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:25.276381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:25.276952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:25.276385 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:25.276952 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:25.276485 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:25.276952 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:25.276595 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:25.276952 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:25.276676 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:27.276294 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:27.276260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:27.276783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:27.276263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:27.276783 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:27.276396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:27.276783 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:27.276497 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:27.276783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:27.276261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:27.276783 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:27.276619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:29.276722 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:29.276688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:29.277133 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:29.276688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:29.277133 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:29.276809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:29.277133 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:29.276802 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:29.277133 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:29.276935 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:29.277133 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:29.277009 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:30.928600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:30.928570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:30.929009 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:30.928704 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:30.929009 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:30.928764 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:46.928749935 +0000 UTC m=+34.118130196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:31.029454 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:31.029420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:31.029600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:31.029467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:31.029600 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029572 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:31.029693 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029623 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:31.029693 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029638 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:31.029693 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029652 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:31.029693 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029640 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:47.02962072 +0000 UTC m=+34.219000984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:31.029693 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.029691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:41:47.029681771 +0000 UTC m=+34.219062036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:31.276101 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:31.276073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:31.276222 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:31.276073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:31.276282 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.276228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:31.276317 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.276301 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:31.276356 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:31.276076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:31.276400 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:31.276382 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:33.277111 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:33.277081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:33.277496 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:33.277219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:33.277496 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:33.277228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:33.277496 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:33.277327 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:33.277496 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:33.277380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:33.277496 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:33.277445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:35.276564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:35.276533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:35.276564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:35.276563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:35.277121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:35.276532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:35.277121 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:35.276666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:35.277121 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:35.276810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:35.277121 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:35.276894 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:36.400378 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.399982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dnm5g" event={"ID":"cf333d5d-2436-4afe-910c-e79c9cd7fa72","Type":"ContainerStarted","Data":"fa9614adc27c3e9f1b467d92fee61ae0d382eaf6ad7ab72adcbec9591dd21b17"} Jul 17 05:41:36.404086 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.404065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" event={"ID":"9dd9f06f-83b5-455a-87c0-0942e531e66c","Type":"ContainerStarted","Data":"1af287a88a9fab7e678488a3a445ea9b477de6015ff83b2bf9b223dc73313260"} Jul 17 05:41:36.405539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.405520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pft2v" event={"ID":"7dbdb628-ab21-484c-9b77-17e81ddd6698","Type":"ContainerStarted","Data":"c422c4859a5ad1bc93170d5839c432d45ca8e4351c50a99c05f146b9d1eb3a70"} Jul 17 05:41:36.407908 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.407891 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:41:36.408266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408235 2574 generic.go:356] "Generic (PLEG): container finished" podID="aafc48fa-52d1-44c9-9b9d-004664557555" containerID="ec0b3fcdb4fd04feabf2783a65a3887110b727ff5cb2877d584ab5b98161df35" exitCode=1 Jul 17 05:41:36.408368 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"d1c06f95ec40930fe1bf1e482da12c5d535354de4614c18cbf493cfaa589f9c8"} Jul 17 05:41:36.408368 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"72e9a0ef3e27334b54346609f5f3fe5ba8548a888c7532c2d2dc2def9a72dbb5"} Jul 17 05:41:36.408497 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"a1538547feeec44e1466af8785a6416f72eac95ebaaab38f3506f6efcd5eba0e"} Jul 17 05:41:36.408497 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerDied","Data":"ec0b3fcdb4fd04feabf2783a65a3887110b727ff5cb2877d584ab5b98161df35"} Jul 17 05:41:36.408497 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.408481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"936ea27317aa6901201d7be48fb69d572550c4b04e75f7f7ded56e1efed4a420"} Jul 17 05:41:36.409764 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.409731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lswjg" event={"ID":"54f7b06a-0ab5-4306-8c29-c0e86a4762cf","Type":"ContainerStarted","Data":"06ffb98df791e6f0d8769ed5dc28f987e074658bf7410dc437826c40f1d19f32"} Jul 17 05:41:36.411140 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.411118 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="907a0b0847e7c548edf90bf953267c6a234a8dc8bbb29c475b8863db2bd99626" exitCode=0 Jul 17 05:41:36.411233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.411197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"907a0b0847e7c548edf90bf953267c6a234a8dc8bbb29c475b8863db2bd99626"} Jul 17 05:41:36.412693 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.412672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" event={"ID":"b18fc67e-4aff-40f1-a2cb-ad954e2829eb","Type":"ContainerStarted","Data":"465cebf00a5a1b50f41ec9357d239cb75cd19a93d7f68bbb002f5388211ba423"} Jul 17 05:41:36.413970 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.413914 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dnm5g" podStartSLOduration=3.418423085 podStartE2EDuration="22.413899913s" totalImagesPullingTime="18.995476828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:36.413319628 +0000 UTC m=+23.602699913" watchObservedRunningTime="2026-07-17 05:41:36.413899913 +0000 UTC m=+23.603280201" Jul 17 05:41:36.414402 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.414380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4lfx" event={"ID":"477a7d2a-8f1a-47ea-9956-0d49528c80f0","Type":"ContainerStarted","Data":"6a2f31f684bf3dd7a703381c928b056be9391385857cb27c7248f80c1c2bd21c"} Jul 17 05:41:36.435932 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.435897 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vs4v5" podStartSLOduration=2.331399485 podStartE2EDuration="22.435885292s" totalImagesPullingTime="20.104485807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:36.435586114 +0000 UTC m=+23.624966397" watchObservedRunningTime="2026-07-17 05:41:36.435885292 +0000 UTC m=+23.625265575" Jul 17 05:41:36.449800 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.449757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lswjg" podStartSLOduration=3.456468524 podStartE2EDuration="22.449747641s" totalImagesPullingTime="18.993279117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:36.449576554 +0000 UTC m=+23.638956839" watchObservedRunningTime="2026-07-17 05:41:36.449747641 +0000 UTC m=+23.639127924" Jul 17 05:41:36.495006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.494953 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pft2v" podStartSLOduration=3.142010867 podStartE2EDuration="22.494939431s" totalImagesPullingTime="19.352928564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:36.494163806 +0000 UTC m=+23.683544099" watchObservedRunningTime="2026-07-17 05:41:36.494939431 +0000 UTC m=+23.684319716" Jul 17 05:41:36.511640 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:36.511591 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r4lfx" podStartSLOduration=3.48062589 podStartE2EDuration="22.511564348s" totalImagesPullingTime="19.030938458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:36.51051292 +0000 UTC m=+23.699893205" watchObservedRunningTime="2026-07-17 05:41:36.511564348 +0000 UTC m=+23.700944632" Jul 17 05:41:37.263900 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.263856 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 05:41:37.276761 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.276739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:37.276864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.276764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:37.276914 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:37.276865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:37.277038 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.277023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:37.277123 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:37.277098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:37.277396 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:37.277368 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:37.419027 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.418983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" event={"ID":"9dd9f06f-83b5-455a-87c0-0942e531e66c","Type":"ContainerStarted","Data":"2490ad5dd9a15772e81c7249df7991f8ef26315b8bcbfe793757b7b37550c8a8"} Jul 17 05:41:37.421542 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.421521 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:41:37.421889 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.421869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"7a8116e645919a31f7d7d6b3c89676004f993e0d7617a7309b1a947695952216"} Jul 17 05:41:37.423068 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.423038 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvdw6" event={"ID":"7f92364b-d158-40b9-927a-b60fa2610dd1","Type":"ContainerStarted","Data":"467a09b2f798c885f69369acdf24650546d9e23fa631c35169733aa917d6d2e1"} Jul 17 05:41:37.439172 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:37.439126 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lvdw6" podStartSLOduration=3.343888888 podStartE2EDuration="23.439113505s" totalImagesPullingTime="20.095224617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:37.438948731 +0000 UTC m=+24.628329013" watchObservedRunningTime="2026-07-17 05:41:37.439113505 +0000 UTC m=+24.628493788" Jul 17 05:41:38.245181 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.245072 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T05:41:37.263873975Z","UUID":"c8a7e3af-be43-4078-96d8-c25c320b1210","Handler":null,"Name":"","Endpoint":""} Jul 17 05:41:38.246706 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.246682 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 05:41:38.246834 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.246719 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 05:41:38.427967 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.427889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" event={"ID":"9dd9f06f-83b5-455a-87c0-0942e531e66c","Type":"ContainerStarted","Data":"8ff8a89dca57900b0639277d2404fd7975938549906d41cbe53cef1bcc665685"} Jul 17 05:41:38.443762 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.443717 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l82vn" podStartSLOduration=3.316967194 podStartE2EDuration="24.443703712s" totalImagesPullingTime="21.126736518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:38.443288905 +0000 UTC m=+25.632669188" watchObservedRunningTime="2026-07-17 05:41:38.443703712 +0000 UTC m=+25.633083995" Jul 17 05:41:38.572347 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.572316 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:38.573050 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:38.573032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:39.276348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.276315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:39.276507 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.276315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:39.276507 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:39.276451 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:39.276612 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.276523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:39.276674 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:39.276645 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:39.276796 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:39.276766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:39.433302 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.433273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:41:39.433748 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.433713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"76bfc52d4b94d7128dafd439f95a5a5fb7f5da10583256d39cc5cf154f63e9c7"} Jul 17 05:41:39.434120 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.434017 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:39.434442 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:39.434428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lswjg" Jul 17 05:41:41.277066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:41.276830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:41.277604 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:41.276828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:41.277604 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:41.277188 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:41.277604 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:41.276889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:41.277604 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:41.277240 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:41.277604 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:41.277361 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:42.444595 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.444564 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:41:42.445163 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.445015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"87d7d2719678db63fa69b534ab74665908e0bb182213a0d8f4d9bda165b3e38a"} Jul 17 05:41:42.445399 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.445377 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:42.445520 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.445505 2574 scope.go:117] "RemoveContainer" containerID="ec0b3fcdb4fd04feabf2783a65a3887110b727ff5cb2877d584ab5b98161df35" Jul 17 05:41:42.448102 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.447413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerStarted","Data":"a2bbb8dba49811b9fb1638ff42575138efaebf52da1a6d51dd241e8638417e4a"} Jul 17 05:41:42.461121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:42.461102 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:43.280218 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.279920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:43.280455 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.280299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:43.280713 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.280690 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:43.280801 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.280786 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:43.280865 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.280846 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:43.280939 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.280910 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:43.451715 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.451685 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="a2bbb8dba49811b9fb1638ff42575138efaebf52da1a6d51dd241e8638417e4a" exitCode=0 Jul 17 05:41:43.452103 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.451761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"a2bbb8dba49811b9fb1638ff42575138efaebf52da1a6d51dd241e8638417e4a"} Jul 17 05:41:43.455440 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.455423 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:41:43.455820 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.455786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" event={"ID":"aafc48fa-52d1-44c9-9b9d-004664557555","Type":"ContainerStarted","Data":"6dcf838f9c7e61515f121a013d66837a490ed080ae6a5770f379075d02e8fcd8"} Jul 17 05:41:43.456135 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.456115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:43.456218 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.456184 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:43.471321 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.471303 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:41:43.500778 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.500726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" podStartSLOduration=9.344020956 podStartE2EDuration="29.500712448s" totalImagesPullingTime="20.156691492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:43.499275121 +0000 UTC m=+30.688655406" watchObservedRunningTime="2026-07-17 05:41:43.500712448 +0000 UTC m=+30.690092730" Jul 17 05:41:43.953017 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.952987 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s4lkz"] Jul 17 05:41:43.953221 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.953082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:43.953221 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.953153 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:43.956033 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.956006 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7rcl"] Jul 17 05:41:43.956143 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.956097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:43.956197 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.956174 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:43.956673 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.956638 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr7tv"] Jul 17 05:41:43.956789 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:43.956737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:43.956878 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:43.956860 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:44.460176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:44.460144 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="b89bdb4bd009dbe42169a8d4e21d216934a2baf2264ea3a04291ef81698a5f45" exitCode=0 Jul 17 05:41:44.460514 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:44.460217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"b89bdb4bd009dbe42169a8d4e21d216934a2baf2264ea3a04291ef81698a5f45"} Jul 17 05:41:45.278100 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:45.278070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:45.278237 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:45.278075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:45.278237 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:45.278209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:45.278237 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:45.278209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:45.278423 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:45.278356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:45.278577 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:45.278540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:45.467522 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:45.467175 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="19e4bda32c89554e394c732a66011795e27410ea46bc4ef0da0c2a42bc9fddc2" exitCode=0 Jul 17 05:41:45.467880 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:45.467327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"19e4bda32c89554e394c732a66011795e27410ea46bc4ef0da0c2a42bc9fddc2"} Jul 17 05:41:46.948864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:46.948629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:46.949374 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:46.949327 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:46.949487 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:46.949423 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:42:18.949399583 +0000 UTC m=+66.138779862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 05:41:47.049934 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:47.049902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:47.050105 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:47.049955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:47.050105 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050061 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:47.050105 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050068 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 05:41:47.050105 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050088 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 05:41:47.050105 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050097 2574 projected.go:196] Error preparing data for projected volume kube-api-access-cg7jc for pod openshift-network-diagnostics/network-check-target-s4lkz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:47.050381 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050134 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret podName:4f55d9ac-02da-4762-8b7d-c1b25fb93f74 nodeName:}" failed. No retries permitted until 2026-07-17 05:42:19.05011461 +0000 UTC m=+66.239494884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret") pod "global-pull-secret-syncer-g7rcl" (UID: "4f55d9ac-02da-4762-8b7d-c1b25fb93f74") : object "kube-system"/"original-pull-secret" not registered Jul 17 05:41:47.050381 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.050155 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc podName:58d666bc-595e-494b-a2cd-d5a060873ef2 nodeName:}" failed. No retries permitted until 2026-07-17 05:42:19.050145213 +0000 UTC m=+66.239525486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cg7jc" (UniqueName: "kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc") pod "network-check-target-s4lkz" (UID: "58d666bc-595e-494b-a2cd-d5a060873ef2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 05:41:47.279810 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:47.279734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:47.279968 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:47.279850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:47.279968 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.279856 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s4lkz" podUID="58d666bc-595e-494b-a2cd-d5a060873ef2" Jul 17 05:41:47.280074 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.279978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:41:47.280074 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:47.280032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:47.280173 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:47.280092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7rcl" podUID="4f55d9ac-02da-4762-8b7d-c1b25fb93f74" Jul 17 05:41:48.653913 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.653835 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeReady" Jul 17 05:41:48.654325 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.653974 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 05:41:48.684222 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.684184 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:41:48.711562 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.711536 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p2l9p"] Jul 17 05:41:48.711783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.711670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.713999 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.713970 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 05:41:48.714118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.714002 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 05:41:48.714118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.714028 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hxf9n\"" Jul 17 05:41:48.714118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.714102 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 05:41:48.721019 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.720645 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 05:41:48.726869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.726847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:41:48.726975 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.726887 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tv449"] Jul 17 05:41:48.726975 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.726962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.728987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.728964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 05:41:48.729084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.729060 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 05:41:48.729320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.729297 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx7lr\"" Jul 17 05:41:48.745491 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.745469 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2l9p"] Jul 17 05:41:48.745623 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.745513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tv449"] Jul 17 05:41:48.745623 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.745601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.747781 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.747756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 05:41:48.747868 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.747756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tmkvs\"" Jul 17 05:41:48.747868 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.747818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 05:41:48.747951 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.747893 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 05:41:48.862134 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljlz\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-tmp-dir\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.862329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.862329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d91c61b2-0c22-4fca-873b-f947862774c5-cert\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.862329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-config-volume\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.862588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzjw\" (UniqueName: \"kubernetes.io/projected/d91c61b2-0c22-4fca-873b-f947862774c5-kube-api-access-7qzjw\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.862830 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqcr5\" (UniqueName: \"kubernetes.io/projected/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-kube-api-access-gqcr5\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.862830 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.862654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.963531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.963531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.963531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.963790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.963790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-config-volume\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.963790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzjw\" (UniqueName: \"kubernetes.io/projected/d91c61b2-0c22-4fca-873b-f947862774c5-kube-api-access-7qzjw\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.963790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqcr5\" (UniqueName: \"kubernetes.io/projected/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-kube-api-access-gqcr5\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.963790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963790 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljlz\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-tmp-dir\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d91c61b2-0c22-4fca-873b-f947862774c5-cert\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964106 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.963986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964454 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:48.964170 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:41:48.964454 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:48.964183 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:41:48.964454 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.964183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.964454 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:48.964261 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:41:49.464230298 +0000 UTC m=+36.653610562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:41:48.964657 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:48.964470 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:48.964657 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:48.964507 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:41:49.46449437 +0000 UTC m=+36.653874635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:41:48.964657 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.964560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-tmp-dir\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.965124 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.965037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-config-volume\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.968669 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.968646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.968766 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.968671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.970622 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.970600 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d91c61b2-0c22-4fca-873b-f947862774c5-cert\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.972711 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.972688 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.974630 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.974606 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqcr5\" (UniqueName: \"kubernetes.io/projected/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-kube-api-access-gqcr5\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:48.974970 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.974949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzjw\" (UniqueName: \"kubernetes.io/projected/d91c61b2-0c22-4fca-873b-f947862774c5-kube-api-access-7qzjw\") pod \"ingress-canary-tv449\" (UID: \"d91c61b2-0c22-4fca-873b-f947862774c5\") " pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:48.975057 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.975040 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:48.975859 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:48.975841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljlz\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:49.054113 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.054076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tv449" Jul 17 05:41:49.276963 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.276887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:41:49.277118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.276887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:41:49.277118 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.277082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:41:49.279589 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.279564 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 05:41:49.279735 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.279587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7x5wz\"" Jul 17 05:41:49.280443 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.280421 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 05:41:49.280560 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.280461 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 05:41:49.280560 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.280475 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jxzr\"" Jul 17 05:41:49.280770 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.280749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 05:41:49.469447 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.469411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:49.469616 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:49.469480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:49.469616 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:49.469579 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:49.469721 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:49.469646 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:41:49.469721 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:49.469668 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:41:49.469721 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:49.469705 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:41:50.469684139 +0000 UTC m=+37.659064401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:41:49.469868 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:49.469724 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:41:50.469717749 +0000 UTC m=+37.659098010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:41:50.478454 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:50.478430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:50.478472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:50.478593 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:50.478599 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:50.478607 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:50.478670 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:41:52.478646215 +0000 UTC m=+39.668026477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:41:50.479058 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:50.478696 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:41:52.47868573 +0000 UTC m=+39.668065995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:41:50.578420 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:50.578227 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tv449"] Jul 17 05:41:50.653899 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:41:50.653867 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91c61b2_0c22_4fca_873b_f947862774c5.slice/crio-2f6ba063df2a6e6e20f42d4b5e455eff59caf2e7a28705eeda32a097cd7c288b WatchSource:0}: Error finding container 2f6ba063df2a6e6e20f42d4b5e455eff59caf2e7a28705eeda32a097cd7c288b: Status 404 returned error can't find the container with id 2f6ba063df2a6e6e20f42d4b5e455eff59caf2e7a28705eeda32a097cd7c288b Jul 17 05:41:51.483783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:51.483746 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="f3bec1b25eedac252cec8c205bf75bfa1bc5df4c8d199ac232e09d4384cd3e52" exitCode=0 Jul 17 05:41:51.484240 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:51.483822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"f3bec1b25eedac252cec8c205bf75bfa1bc5df4c8d199ac232e09d4384cd3e52"} Jul 17 05:41:51.485059 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:51.485036 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tv449" event={"ID":"d91c61b2-0c22-4fca-873b-f947862774c5","Type":"ContainerStarted","Data":"2f6ba063df2a6e6e20f42d4b5e455eff59caf2e7a28705eeda32a097cd7c288b"} Jul 17 05:41:52.492209 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:52.492170 2574 generic.go:356] "Generic (PLEG): container finished" podID="3df423e1-e9ed-45bc-ac94-f443e1686291" containerID="698a528b036a9e013a5a352ee3e30ba9c1545c053be2e064b51e96dafb1e90d4" exitCode=0 Jul 17 05:41:52.492720 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:52.492227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerDied","Data":"698a528b036a9e013a5a352ee3e30ba9c1545c053be2e064b51e96dafb1e90d4"} Jul 17 05:41:52.494295 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:52.494273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:52.494401 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:52.494327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:52.494450 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:52.494426 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:41:52.494450 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:52.494444 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:41:52.494510 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:52.494502 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:41:56.494482971 +0000 UTC m=+43.683863234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:41:52.494781 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:52.494423 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:52.494831 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:52.494810 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:41:56.494796778 +0000 UTC m=+43.684177039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:41:53.497867 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:53.497699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9flj" event={"ID":"3df423e1-e9ed-45bc-ac94-f443e1686291","Type":"ContainerStarted","Data":"fcb6633fcf1669fac127c86d25bb8e31f8b5dcd361f7785a53228a310661a3d2"} Jul 17 05:41:53.519675 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:53.519632 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9flj" podStartSLOduration=7.804217782 podStartE2EDuration="39.519617687s" totalImagesPullingTime="31.715399905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:53.518406692 +0000 UTC m=+40.707786975" watchObservedRunningTime="2026-07-17 05:41:53.519617687 +0000 UTC m=+40.708997970" Jul 17 05:41:54.501535 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:54.501498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tv449" event={"ID":"d91c61b2-0c22-4fca-873b-f947862774c5","Type":"ContainerStarted","Data":"77542ccf4ae620ab93da95622bbc95ec1cc912123356fd89de0252991bc559fd"} Jul 17 05:41:56.525927 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:56.525893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:41:56.525961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:56.526052 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:56.526063 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:56.526084 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:56.526102 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:42:04.52608659 +0000 UTC m=+51.715466871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:41:56.526377 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:41:56.526133 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:42:04.526116376 +0000 UTC m=+51.715496637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:42:04.580468 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:04.580429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:04.580480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:04.580577 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:04.580582 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:04.580637 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:42:20.58062149 +0000 UTC m=+67.770001750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:04.580649 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:42:04.580916 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:04.580703 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:42:20.580691616 +0000 UTC m=+67.770071877 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:42:15.478948 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:15.478922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lgqkw" Jul 17 05:42:15.504213 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:15.504168 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tv449" podStartSLOduration=24.639035889 podStartE2EDuration="27.504150248s" totalImagesPullingTime="2.865114359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:41:54.516649577 +0000 UTC m=+41.706029862" watchObservedRunningTime="2026-07-17 05:42:15.504150248 +0000 UTC m=+62.693530534" Jul 17 05:42:18.982672 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:18.982636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:42:18.984867 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:18.984848 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 05:42:18.993839 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:18.993816 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:42:18.993892 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:18.993875 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:43:22.993859882 +0000 UTC m=+130.183240147 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : secret "metrics-daemon-secret" not found Jul 17 05:42:19.083999 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.083961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:42:19.084109 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.084053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:42:19.087037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.087021 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 05:42:19.087085 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.087043 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 05:42:19.097374 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.097346 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 05:42:19.098698 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.098682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4f55d9ac-02da-4762-8b7d-c1b25fb93f74-original-pull-secret\") pod \"global-pull-secret-syncer-g7rcl\" (UID: \"4f55d9ac-02da-4762-8b7d-c1b25fb93f74\") " pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:42:19.107210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.107193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7jc\" (UniqueName: \"kubernetes.io/projected/58d666bc-595e-494b-a2cd-d5a060873ef2-kube-api-access-cg7jc\") pod \"network-check-target-s4lkz\" (UID: \"58d666bc-595e-494b-a2cd-d5a060873ef2\") " pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:42:19.297304 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.297226 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jxzr\"" Jul 17 05:42:19.301329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.301314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7rcl" Jul 17 05:42:19.305455 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.305440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:42:19.458683 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.458657 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7rcl"] Jul 17 05:42:19.461560 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:42:19.461529 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f55d9ac_02da_4762_8b7d_c1b25fb93f74.slice/crio-d6c6ba95efe93fb50dc7c6eb876724fe3efa555cad5ba718389fed77b24e1dd7 WatchSource:0}: Error finding container d6c6ba95efe93fb50dc7c6eb876724fe3efa555cad5ba718389fed77b24e1dd7: Status 404 returned error can't find the container with id d6c6ba95efe93fb50dc7c6eb876724fe3efa555cad5ba718389fed77b24e1dd7 Jul 17 05:42:19.468820 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.468797 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s4lkz"] Jul 17 05:42:19.472527 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:42:19.472505 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d666bc_595e_494b_a2cd_d5a060873ef2.slice/crio-7afa2802228b150e0b80b04657e14d2d9a028905421bfb4b05f125f6cf1e6632 WatchSource:0}: Error finding container 7afa2802228b150e0b80b04657e14d2d9a028905421bfb4b05f125f6cf1e6632: Status 404 returned error can't find the container with id 7afa2802228b150e0b80b04657e14d2d9a028905421bfb4b05f125f6cf1e6632 Jul 17 05:42:19.561325 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.561243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7rcl" event={"ID":"4f55d9ac-02da-4762-8b7d-c1b25fb93f74","Type":"ContainerStarted","Data":"d6c6ba95efe93fb50dc7c6eb876724fe3efa555cad5ba718389fed77b24e1dd7"} Jul 17 05:42:19.562364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:19.562343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s4lkz" event={"ID":"58d666bc-595e-494b-a2cd-d5a060873ef2","Type":"ContainerStarted","Data":"7afa2802228b150e0b80b04657e14d2d9a028905421bfb4b05f125f6cf1e6632"} Jul 17 05:42:20.595783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:20.595743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:20.595841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:20.595925 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:20.595948 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:20.595952 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:20.596035 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:42:52.59601499 +0000 UTC m=+99.785395268 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:42:20.596275 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:20.596054 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:42:52.596044599 +0000 UTC m=+99.785424866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:42:24.578092 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:24.578060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s4lkz" event={"ID":"58d666bc-595e-494b-a2cd-d5a060873ef2","Type":"ContainerStarted","Data":"87356c907e0db273a0bde653ddf76731466262e452a3bccd83f60328d6dd0add"} Jul 17 05:42:24.578538 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:24.578130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:42:24.579665 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:24.579628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7rcl" event={"ID":"4f55d9ac-02da-4762-8b7d-c1b25fb93f74","Type":"ContainerStarted","Data":"686f7f2ee4d8e3926f815f772bc1bef6d6736f877d347c1949e11010dc6621fa"} Jul 17 05:42:24.592361 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:24.592312 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s4lkz" podStartSLOduration=65.807636107 podStartE2EDuration="1m10.592294425s" totalImagesPullingTime="4.784658318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:24.590546308 +0000 UTC m=+71.779926592" watchObservedRunningTime="2026-07-17 05:42:24.592294425 +0000 UTC m=+71.781674708" Jul 17 05:42:24.606789 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:24.606741 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g7rcl" podStartSLOduration=65.807101898 podStartE2EDuration="1m10.606725806s" totalImagesPullingTime="4.799623908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:42:24.606611591 +0000 UTC m=+71.795991875" watchObservedRunningTime="2026-07-17 05:42:24.606725806 +0000 UTC m=+71.796106099" Jul 17 05:42:52.616861 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:52.616807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:42:52.616861 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:52.616869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:42:52.617624 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:52.616969 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:42:52.617624 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:52.616991 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:42:52.617624 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:52.616966 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:42:52.617624 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:52.617069 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:43:56.617053234 +0000 UTC m=+163.806433495 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:42:52.617624 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:42:52.617085 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:43:56.617078887 +0000 UTC m=+163.806459148 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:42:55.584736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:42:55.584709 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s4lkz" Jul 17 05:43:23.025590 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:43:23.025544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:43:23.026087 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:23.025689 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:43:23.026087 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:23.025759 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:25.025740107 +0000 UTC m=+252.215120388 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : secret "metrics-daemon-secret" not found Jul 17 05:43:51.724581 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:51.724544 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" Jul 17 05:43:51.736637 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:51.736602 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p2l9p" podUID="aa8fcf25-18bd-42b7-bc0b-bbd34870577a" Jul 17 05:43:51.785479 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:43:51.785457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:43:51.785600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:43:51.785457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:43:52.288915 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:52.288882 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:43:56.655177 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:43:56.655142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:43:56.655201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:56.655294 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:56.655297 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:56.655313 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d99669ff8-c62jl: secret "image-registry-tls" not found Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:56.655338 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls podName:aa8fcf25-18bd-42b7-bc0b-bbd34870577a nodeName:}" failed. No retries permitted until 2026-07-17 05:45:58.655325206 +0000 UTC m=+285.844705466 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls") pod "dns-default-p2l9p" (UID: "aa8fcf25-18bd-42b7-bc0b-bbd34870577a") : secret "dns-default-metrics-tls" not found Jul 17 05:43:56.655585 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:43:56.655359 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls podName:fd1f5f1d-a4bf-4acd-8594-ca17cb10750e nodeName:}" failed. No retries permitted until 2026-07-17 05:45:58.655347417 +0000 UTC m=+285.844727678 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls") pod "image-registry-7d99669ff8-c62jl" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e") : secret "image-registry-tls" not found Jul 17 05:44:03.277314 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:44:03.277280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:45:15.466476 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.466441 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr"] Jul 17 05:45:15.468388 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.468362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" Jul 17 05:45:15.472680 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.472656 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b96pq\"" Jul 17 05:45:15.476269 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.476226 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-m5wkv"] Jul 17 05:45:15.477869 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.477850 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l"] Jul 17 05:45:15.477952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.477934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.479763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.479740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.480205 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.480186 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bmkpk\"" Jul 17 05:45:15.483894 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.483875 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 05:45:15.484030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.483999 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 05:45:15.484229 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.484074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 17 05:45:15.484229 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.484010 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 17 05:45:15.484812 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.484793 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:45:15.485179 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.485059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 17 05:45:15.485179 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.485093 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9mckd\"" Jul 17 05:45:15.485179 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.485113 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 17 05:45:15.485408 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.485202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr"] Jul 17 05:45:15.489159 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.489134 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 17 05:45:15.491169 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.491139 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-m5wkv"] Jul 17 05:45:15.493692 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.493671 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l"] Jul 17 05:45:15.562973 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.562945 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n"] Jul 17 05:45:15.564775 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.564756 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" Jul 17 05:45:15.567034 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.567006 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn"] Jul 17 05:45:15.567131 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.567099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6v6cn\"" Jul 17 05:45:15.567436 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.567417 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:45:15.567797 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.567781 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 17 05:45:15.570600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.570580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.570678 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.570581 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-687684fd5b-qjmnh"] Jul 17 05:45:15.572809 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.572793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.573239 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.573224 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 05:45:15.573344 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.573238 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 17 05:45:15.573344 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.573241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7djgd\"" Jul 17 05:45:15.574633 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.574610 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 05:45:15.574633 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.574619 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 17 05:45:15.574851 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.574838 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-98chg\"" Jul 17 05:45:15.576609 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.576592 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 17 05:45:15.576708 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.576612 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 17 05:45:15.576708 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.576634 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 17 05:45:15.576855 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.576840 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 17 05:45:15.576919 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.576845 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 05:45:15.577121 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.577105 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 05:45:15.577318 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.577302 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n"] Jul 17 05:45:15.580330 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.580312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn"] Jul 17 05:45:15.582091 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-service-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.582266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hb9h\" (UniqueName: \"kubernetes.io/projected/fcadf004-328f-4c25-bd51-34074c28ea9e-kube-api-access-6hb9h\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.582394 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.582485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkclb\" (UniqueName: \"kubernetes.io/projected/ac219b5a-2344-4ad3-8b12-0ddebb7895b7-kube-api-access-tkclb\") pod \"network-check-source-7bc5b6bf85-jhllr\" (UID: \"ac219b5a-2344-4ad3-8b12-0ddebb7895b7\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" Jul 17 05:45:15.582485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd4s\" (UniqueName: \"kubernetes.io/projected/23848f9a-f634-4192-8ef2-d18b0061e453-kube-api-access-vkd4s\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.582485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-tmp\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.582644 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.582644 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-snapshots\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.582644 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.582603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcadf004-328f-4c25-bd51-34074c28ea9e-serving-cert\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.587595 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.587575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-687684fd5b-qjmnh"] Jul 17 05:45:15.683603 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-stats-auth\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.683767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-tmp\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.683767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-snapshots\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.683767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcadf004-328f-4c25-bd51-34074c28ea9e-serving-cert\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.683767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-default-certificate\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.683767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-service-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hb9h\" (UniqueName: \"kubernetes.io/projected/fcadf004-328f-4c25-bd51-34074c28ea9e-kube-api-access-6hb9h\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/66ede3fa-03c0-4bfc-b099-865b673591ec-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkclb\" (UniqueName: \"kubernetes.io/projected/ac219b5a-2344-4ad3-8b12-0ddebb7895b7-kube-api-access-tkclb\") pod \"network-check-source-7bc5b6bf85-jhllr\" (UID: \"ac219b5a-2344-4ad3-8b12-0ddebb7895b7\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdnv\" (UniqueName: \"kubernetes.io/projected/b2c9e7f3-a354-47d7-9c89-9879813f0705-kube-api-access-cvdnv\") pod \"volume-data-source-validator-78cf4c6f6-gp55n\" (UID: \"b2c9e7f3-a354-47d7-9c89-9879813f0705\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd4s\" (UniqueName: \"kubernetes.io/projected/23848f9a-f634-4192-8ef2-d18b0061e453-kube-api-access-vkd4s\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.684030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.683992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkg5\" (UniqueName: \"kubernetes.io/projected/66ede3fa-03c0-4bfc-b099-865b673591ec-kube-api-access-rzkg5\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-tmp\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f884\" (UniqueName: \"kubernetes.io/projected/bef8682f-58cd-475a-a989-97a2a548fce0-kube-api-access-7f884\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.684162 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.684229 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls podName:23848f9a-f634-4192-8ef2-d18b0061e453 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:16.184209523 +0000 UTC m=+243.373589798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qpz2l" (UID: "23848f9a-f634-4192-8ef2-d18b0061e453") : secret "samples-operator-tls" not found Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684340 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/fcadf004-328f-4c25-bd51-34074c28ea9e-snapshots\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.684432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.684353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-service-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.685054 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.685035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcadf004-328f-4c25-bd51-34074c28ea9e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.686073 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.686056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fcadf004-328f-4c25-bd51-34074c28ea9e-serving-cert\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.697984 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.697957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hb9h\" (UniqueName: \"kubernetes.io/projected/fcadf004-328f-4c25-bd51-34074c28ea9e-kube-api-access-6hb9h\") pod \"insights-operator-555d9d7885-m5wkv\" (UID: \"fcadf004-328f-4c25-bd51-34074c28ea9e\") " pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.698154 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.698131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkclb\" (UniqueName: \"kubernetes.io/projected/ac219b5a-2344-4ad3-8b12-0ddebb7895b7-kube-api-access-tkclb\") pod \"network-check-source-7bc5b6bf85-jhllr\" (UID: \"ac219b5a-2344-4ad3-8b12-0ddebb7895b7\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" Jul 17 05:45:15.698618 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.698602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd4s\" (UniqueName: \"kubernetes.io/projected/23848f9a-f634-4192-8ef2-d18b0061e453-kube-api-access-vkd4s\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:15.778054 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.777991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" Jul 17 05:45:15.784753 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkg5\" (UniqueName: \"kubernetes.io/projected/66ede3fa-03c0-4bfc-b099-865b673591ec-kube-api-access-rzkg5\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.784850 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f884\" (UniqueName: \"kubernetes.io/projected/bef8682f-58cd-475a-a989-97a2a548fce0-kube-api-access-7f884\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.784850 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.784850 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-stats-auth\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.784850 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-default-certificate\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.785055 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.785055 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/66ede3fa-03c0-4bfc-b099-865b673591ec-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.785055 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.785055 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.784990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdnv\" (UniqueName: \"kubernetes.io/projected/b2c9e7f3-a354-47d7-9c89-9879813f0705-kube-api-access-cvdnv\") pod \"volume-data-source-validator-78cf4c6f6-gp55n\" (UID: \"b2c9e7f3-a354-47d7-9c89-9879813f0705\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" Jul 17 05:45:15.785055 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.784996 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 05:45:15.785311 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.785062 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:16.28504422 +0000 UTC m=+243.474424493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : secret "router-metrics-certs-default" not found Jul 17 05:45:15.785311 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.785107 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:16.285089331 +0000 UTC m=+243.474469597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:15.785527 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.785507 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:15.785652 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:15.785562 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:16.285545155 +0000 UTC m=+243.474925421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:15.785895 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.785874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/66ede3fa-03c0-4bfc-b099-865b673591ec-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.787446 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.787424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-stats-auth\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.787727 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.787703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-default-certificate\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.787846 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.787794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" Jul 17 05:45:15.796373 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.796353 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f884\" (UniqueName: \"kubernetes.io/projected/bef8682f-58cd-475a-a989-97a2a548fce0-kube-api-access-7f884\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:15.796454 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.796400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkg5\" (UniqueName: \"kubernetes.io/projected/66ede3fa-03c0-4bfc-b099-865b673591ec-kube-api-access-rzkg5\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:15.796600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.796541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdnv\" (UniqueName: \"kubernetes.io/projected/b2c9e7f3-a354-47d7-9c89-9879813f0705-kube-api-access-cvdnv\") pod \"volume-data-source-validator-78cf4c6f6-gp55n\" (UID: \"b2c9e7f3-a354-47d7-9c89-9879813f0705\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" Jul 17 05:45:15.880278 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.880212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" Jul 17 05:45:15.912855 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.912823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr"] Jul 17 05:45:15.915764 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:15.915740 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac219b5a_2344_4ad3_8b12_0ddebb7895b7.slice/crio-b10e6ac5dab8a1413736128c0b757b5a08622b87570ed9e76533aa6b67c3c521 WatchSource:0}: Error finding container b10e6ac5dab8a1413736128c0b757b5a08622b87570ed9e76533aa6b67c3c521: Status 404 returned error can't find the container with id b10e6ac5dab8a1413736128c0b757b5a08622b87570ed9e76533aa6b67c3c521 Jul 17 05:45:15.929172 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.929139 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-m5wkv"] Jul 17 05:45:15.984461 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.984429 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" event={"ID":"fcadf004-328f-4c25-bd51-34074c28ea9e","Type":"ContainerStarted","Data":"856fc4020477b9f663710769ab0d690ba6b6f06f2871dfeb862f192e3d085f74"} Jul 17 05:45:15.986279 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.986158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" event={"ID":"ac219b5a-2344-4ad3-8b12-0ddebb7895b7","Type":"ContainerStarted","Data":"efc1f14510e6565d2a2bb07cf682f0e2924169031d0cd13ef2b93a465b015db7"} Jul 17 05:45:15.986279 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:15.986207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" event={"ID":"ac219b5a-2344-4ad3-8b12-0ddebb7895b7","Type":"ContainerStarted","Data":"b10e6ac5dab8a1413736128c0b757b5a08622b87570ed9e76533aa6b67c3c521"} Jul 17 05:45:16.004802 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.004757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-jhllr" podStartSLOduration=1.004741549 podStartE2EDuration="1.004741549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:16.004169454 +0000 UTC m=+243.193549737" watchObservedRunningTime="2026-07-17 05:45:16.004741549 +0000 UTC m=+243.194121832" Jul 17 05:45:16.015142 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.015117 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n"] Jul 17 05:45:16.017777 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:16.017752 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c9e7f3_a354_47d7_9c89_9879813f0705.slice/crio-e63d8896ca7898f3e58cdaac16a9e335abf7f8c9ce30533e3aee0c45cf47606a WatchSource:0}: Error finding container e63d8896ca7898f3e58cdaac16a9e335abf7f8c9ce30533e3aee0c45cf47606a: Status 404 returned error can't find the container with id e63d8896ca7898f3e58cdaac16a9e335abf7f8c9ce30533e3aee0c45cf47606a Jul 17 05:45:16.188001 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.187967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:16.188192 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.188121 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 17 05:45:16.188192 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.188182 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls podName:23848f9a-f634-4192-8ef2-d18b0061e453 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:17.188167586 +0000 UTC m=+244.377547847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qpz2l" (UID: "23848f9a-f634-4192-8ef2-d18b0061e453") : secret "samples-operator-tls" not found Jul 17 05:45:16.288685 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.288649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:16.288864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.288713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:16.288864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.288784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:16.288864 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.288787 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:16.288864 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.288828 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 05:45:16.289015 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.288879 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:17.288863865 +0000 UTC m=+244.478244126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:16.289015 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.288897 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:17.288891267 +0000 UTC m=+244.478271527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:16.289015 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:16.288912 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:17.288904447 +0000 UTC m=+244.478284708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : secret "router-metrics-certs-default" not found Jul 17 05:45:16.990660 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:16.990590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" event={"ID":"b2c9e7f3-a354-47d7-9c89-9879813f0705","Type":"ContainerStarted","Data":"e63d8896ca7898f3e58cdaac16a9e335abf7f8c9ce30533e3aee0c45cf47606a"} Jul 17 05:45:17.196792 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.196728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:17.196983 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.196906 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 17 05:45:17.197062 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.196995 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls podName:23848f9a-f634-4192-8ef2-d18b0061e453 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.196969788 +0000 UTC m=+246.386350062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qpz2l" (UID: "23848f9a-f634-4192-8ef2-d18b0061e453") : secret "samples-operator-tls" not found Jul 17 05:45:17.297561 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.297492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:17.297561 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.297551 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.297605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.297661 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.297706 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.297729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.297710094 +0000 UTC m=+246.487090370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : secret "router-metrics-certs-default" not found Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.297749 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.297738775 +0000 UTC m=+246.487119047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:17.297767 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:17.297764 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.29775582 +0000 UTC m=+246.487136093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:17.996482 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.996449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" event={"ID":"fcadf004-328f-4c25-bd51-34074c28ea9e","Type":"ContainerStarted","Data":"2485ff843784b6cbd039f3cace06c01e879cc2a4078411bb9bd5a110b6e05add"} Jul 17 05:45:17.997791 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:17.997767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" event={"ID":"b2c9e7f3-a354-47d7-9c89-9879813f0705","Type":"ContainerStarted","Data":"d3d1d82e8047d954f14e2958ce65154b6fa3511c2599d36e268c3ab304ee50fd"} Jul 17 05:45:18.011828 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.011785 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" podStartSLOduration=1.012743723 podStartE2EDuration="3.011771015s" totalImagesPullingTime="1.999027292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:18.011072843 +0000 UTC m=+245.200453136" watchObservedRunningTime="2026-07-17 05:45:18.011771015 +0000 UTC m=+245.201151298" Jul 17 05:45:18.026129 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.026084 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gp55n" podStartSLOduration=1.7370361600000002 podStartE2EDuration="3.026067504s" totalImagesPullingTime="1.289031344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:18.025273522 +0000 UTC m=+245.214653803" watchObservedRunningTime="2026-07-17 05:45:18.026067504 +0000 UTC m=+245.215447784" Jul 17 05:45:18.938579 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.938542 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cv9vc"] Jul 17 05:45:18.940868 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.940844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:18.943375 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.943352 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 05:45:18.943513 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.943396 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 05:45:18.943513 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.943407 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8zw9n\"" Jul 17 05:45:18.955054 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:18.955036 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cv9vc"] Jul 17 05:45:19.113372 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.113339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.113769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.113401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63a926d0-adb4-453a-b7c1-568f38da49f2-crio-socket\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.113769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.113438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.113769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.113501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a926d0-adb4-453a-b7c1-568f38da49f2-data-volume\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.113769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.113636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vxv\" (UniqueName: \"kubernetes.io/projected/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-api-access-b8vxv\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214187 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63a926d0-adb4-453a-b7c1-568f38da49f2-crio-socket\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214187 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:19.214187 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a926d0-adb4-453a-b7c1-568f38da49f2-data-volume\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63a926d0-adb4-453a-b7c1-568f38da49f2-crio-socket\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214312 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214324 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214373 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls podName:23848f9a-f634-4192-8ef2-d18b0061e453 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:23.214354505 +0000 UTC m=+250.403734766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qpz2l" (UID: "23848f9a-f634-4192-8ef2-d18b0061e453") : secret "samples-operator-tls" not found Jul 17 05:45:19.214451 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vxv\" (UniqueName: \"kubernetes.io/projected/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-api-access-b8vxv\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214686 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214686 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.714505124 +0000 UTC m=+246.903885386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.214686 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214544 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:19.214686 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.214548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63a926d0-adb4-453a-b7c1-568f38da49f2-data-volume\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.214686 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.214580 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:19.714568837 +0000 UTC m=+246.903949112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : configmap "kube-rbac-proxy" not found Jul 17 05:45:19.221983 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.221959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vxv\" (UniqueName: \"kubernetes.io/projected/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-api-access-b8vxv\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.315697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.315671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:19.315829 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.315732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:19.315829 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.315807 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:19.315911 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.315844 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:23.315829926 +0000 UTC m=+250.505210201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:19.315911 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.315859 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:23.315852564 +0000 UTC m=+250.505232833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:19.315991 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.315910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:19.316040 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.315991 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 05:45:19.316040 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.316028 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:23.316012286 +0000 UTC m=+250.505392549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : secret "router-metrics-certs-default" not found Jul 17 05:45:19.718647 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.718605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.718789 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:19.718696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:19.718789 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.718729 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:19.718875 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.718794 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:20.718779469 +0000 UTC m=+247.908159733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : configmap "kube-rbac-proxy" not found Jul 17 05:45:19.718875 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.718806 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:19.718875 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:19.718850 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:20.718833829 +0000 UTC m=+247.908214091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:20.726141 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:20.726111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:20.726611 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:20.726169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:20.726611 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:20.726235 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:20.726611 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:20.726274 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:20.726611 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:20.726320 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:22.7263038 +0000 UTC m=+249.915684074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : configmap "kube-rbac-proxy" not found Jul 17 05:45:20.726611 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:20.726335 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:22.726329147 +0000 UTC m=+249.915709412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:20.826976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:20.826948 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dnm5g_cf333d5d-2436-4afe-910c-e79c9cd7fa72/dns-node-resolver/0.log" Jul 17 05:45:22.027529 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:22.027501 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pft2v_7dbdb628-ab21-484c-9b77-17e81ddd6698/node-ca/0.log" Jul 17 05:45:22.740920 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:22.740880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:22.741096 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:22.740970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:22.741096 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:22.740980 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:22.741096 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:22.741036 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:26.741020997 +0000 UTC m=+253.930401258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:22.741096 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:22.741046 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:22.741096 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:22.741093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:26.741078316 +0000 UTC m=+253.930458580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : configmap "kube-rbac-proxy" not found Jul 17 05:45:23.027493 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:23.027409 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tv449_d91c61b2-0c22-4fca-873b-f947862774c5/serve-healthcheck-canary/0.log" Jul 17 05:45:23.244483 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:23.244439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:23.244875 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.244591 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 17 05:45:23.244875 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.244682 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls podName:23848f9a-f634-4192-8ef2-d18b0061e453 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:31.244666998 +0000 UTC m=+258.434047259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-qpz2l" (UID: "23848f9a-f634-4192-8ef2-d18b0061e453") : secret "samples-operator-tls" not found Jul 17 05:45:23.345178 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:23.345144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:23.345356 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:23.345205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:23.345356 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:23.345301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:23.345428 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.345348 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:23.345428 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.345386 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:31.345366104 +0000 UTC m=+258.534746369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:23.345428 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.345392 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 05:45:23.345428 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.345414 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:31.345397808 +0000 UTC m=+258.534778073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:23.345557 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:23.345435 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:31.345424978 +0000 UTC m=+258.534805240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : secret "router-metrics-certs-default" not found Jul 17 05:45:25.059734 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:25.059705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:45:25.060132 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:25.059805 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 05:45:25.060132 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:25.059852 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs podName:6617226d-ebd9-40ab-ae75-4fec0d433e35 nodeName:}" failed. No retries permitted until 2026-07-17 05:47:27.059839286 +0000 UTC m=+374.249219548 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs") pod "network-metrics-daemon-gr7tv" (UID: "6617226d-ebd9-40ab-ae75-4fec0d433e35") : secret "metrics-daemon-secret" not found Jul 17 05:45:26.773134 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:26.773090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:26.773615 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:26.773190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:26.773615 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:26.773230 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 05:45:26.773615 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:26.773310 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:34.773294154 +0000 UTC m=+261.962674415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : configmap "kube-rbac-proxy" not found Jul 17 05:45:26.773615 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:26.773356 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:26.773615 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:26.773415 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:34.773399253 +0000 UTC m=+261.962779517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:31.310286 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.310235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:31.312751 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.312722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/23848f9a-f634-4192-8ef2-d18b0061e453-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-qpz2l\" (UID: \"23848f9a-f634-4192-8ef2-d18b0061e453\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:31.394037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.394015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" Jul 17 05:45:31.411519 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.411493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:31.411637 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.411553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:31.411637 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.411626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:31.411742 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:31.411656 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:31.411742 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:31.411735 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls podName:66ede3fa-03c0-4bfc-b099-865b673591ec nodeName:}" failed. No retries permitted until 2026-07-17 05:45:47.411715282 +0000 UTC m=+274.601095557 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-bhndn" (UID: "66ede3fa-03c0-4bfc-b099-865b673591ec") : secret "cluster-monitoring-operator-tls" not found Jul 17 05:45:31.411829 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:31.411761 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle podName:bef8682f-58cd-475a-a989-97a2a548fce0 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:47.411742289 +0000 UTC m=+274.601122569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle") pod "router-default-687684fd5b-qjmnh" (UID: "bef8682f-58cd-475a-a989-97a2a548fce0") : configmap references non-existent config key: service-ca.crt Jul 17 05:45:31.413790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.413772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bef8682f-58cd-475a-a989-97a2a548fce0-metrics-certs\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:31.516538 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:31.516510 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l"] Jul 17 05:45:32.038985 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:32.038949 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" event={"ID":"23848f9a-f634-4192-8ef2-d18b0061e453","Type":"ContainerStarted","Data":"2e14cb9353047a146403cd5e303cb6196a19c625c9d19a3e35c0360b8634e6d4"} Jul 17 05:45:34.045976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.045902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" event={"ID":"23848f9a-f634-4192-8ef2-d18b0061e453","Type":"ContainerStarted","Data":"24cb7f3db75420e6a8093b47f65f26d83bfd707fea88703dfcee8aa590e6c8f4"} Jul 17 05:45:34.045976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.045952 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" event={"ID":"23848f9a-f634-4192-8ef2-d18b0061e453","Type":"ContainerStarted","Data":"b41a9a5142f7934f6b4ca3fc3c62b139a22b589cba511d4271ed1bd96763bc02"} Jul 17 05:45:34.061439 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.061391 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-qpz2l" podStartSLOduration=17.346136184 podStartE2EDuration="19.061356272s" totalImagesPullingTime="1.715220088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:34.060515443 +0000 UTC m=+261.249895725" watchObservedRunningTime="2026-07-17 05:45:34.061356272 +0000 UTC m=+261.250736605" Jul 17 05:45:34.840523 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.840490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:34.840718 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.840549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:34.840718 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:34.840660 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 05:45:34.840718 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:34.840718 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls podName:63a926d0-adb4-453a-b7c1-568f38da49f2 nodeName:}" failed. No retries permitted until 2026-07-17 05:45:50.840705337 +0000 UTC m=+278.030085597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cv9vc" (UID: "63a926d0-adb4-453a-b7c1-568f38da49f2") : secret "insights-runtime-extractor-tls" not found Jul 17 05:45:34.841037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:34.841018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63a926d0-adb4-453a-b7c1-568f38da49f2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:41.904410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.904380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-wmplw"] Jul 17 05:45:41.906854 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.906840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:41.909398 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.909374 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 05:45:41.909501 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.909407 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mbz4b\"" Jul 17 05:45:41.909578 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.909516 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 05:45:41.917499 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.917451 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wmplw"] Jul 17 05:45:41.994607 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:41.994575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5bhs\" (UniqueName: \"kubernetes.io/projected/b785f4f1-6406-434e-8376-e74896762cd5-kube-api-access-b5bhs\") pod \"downloads-5bc67dcd46-wmplw\" (UID: \"b785f4f1-6406-434e-8376-e74896762cd5\") " pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:42.095078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:42.095041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5bhs\" (UniqueName: \"kubernetes.io/projected/b785f4f1-6406-434e-8376-e74896762cd5-kube-api-access-b5bhs\") pod \"downloads-5bc67dcd46-wmplw\" (UID: \"b785f4f1-6406-434e-8376-e74896762cd5\") " pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:42.103077 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:42.103051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5bhs\" (UniqueName: \"kubernetes.io/projected/b785f4f1-6406-434e-8376-e74896762cd5-kube-api-access-b5bhs\") pod \"downloads-5bc67dcd46-wmplw\" (UID: \"b785f4f1-6406-434e-8376-e74896762cd5\") " pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:42.215857 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:42.215765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:42.341303 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:42.341244 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wmplw"] Jul 17 05:45:42.344522 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:42.344498 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb785f4f1_6406_434e_8376_e74896762cd5.slice/crio-90583025108aa5e2085abc35447f6963c286a9f15434ad68a14363cd72d20853 WatchSource:0}: Error finding container 90583025108aa5e2085abc35447f6963c286a9f15434ad68a14363cd72d20853: Status 404 returned error can't find the container with id 90583025108aa5e2085abc35447f6963c286a9f15434ad68a14363cd72d20853 Jul 17 05:45:43.073897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:43.073857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wmplw" event={"ID":"b785f4f1-6406-434e-8376-e74896762cd5","Type":"ContainerStarted","Data":"90583025108aa5e2085abc35447f6963c286a9f15434ad68a14363cd72d20853"} Jul 17 05:45:47.442742 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.442698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:47.443239 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.442824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:47.443446 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.443418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bef8682f-58cd-475a-a989-97a2a548fce0-service-ca-bundle\") pod \"router-default-687684fd5b-qjmnh\" (UID: \"bef8682f-58cd-475a-a989-97a2a548fce0\") " pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:47.445633 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.445613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/66ede3fa-03c0-4bfc-b099-865b673591ec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-bhndn\" (UID: \"66ede3fa-03c0-4bfc-b099-865b673591ec\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:47.686261 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.686211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" Jul 17 05:45:47.692103 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.692068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:47.835589 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.835553 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn"] Jul 17 05:45:47.839100 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:47.839066 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ede3fa_03c0_4bfc_b099_865b673591ec.slice/crio-c4d1a56a89c30f38ce6ebc0ccf9947c5adcee2f951edd58c13bbecc424777484 WatchSource:0}: Error finding container c4d1a56a89c30f38ce6ebc0ccf9947c5adcee2f951edd58c13bbecc424777484: Status 404 returned error can't find the container with id c4d1a56a89c30f38ce6ebc0ccf9947c5adcee2f951edd58c13bbecc424777484 Jul 17 05:45:47.861684 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:47.861658 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-687684fd5b-qjmnh"] Jul 17 05:45:47.864632 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:47.864605 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef8682f_58cd_475a_a989_97a2a548fce0.slice/crio-f8e6f3e234f45d5dfe9e651c913032438905f0ed917400e88b371a7c953e992f WatchSource:0}: Error finding container f8e6f3e234f45d5dfe9e651c913032438905f0ed917400e88b371a7c953e992f: Status 404 returned error can't find the container with id f8e6f3e234f45d5dfe9e651c913032438905f0ed917400e88b371a7c953e992f Jul 17 05:45:48.094181 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.094056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-687684fd5b-qjmnh" event={"ID":"bef8682f-58cd-475a-a989-97a2a548fce0","Type":"ContainerStarted","Data":"7678c6ec163c1a7300e920bf476acf0202d4c31393da412b5a1aa4cde64d7bbe"} Jul 17 05:45:48.094402 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.094151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-687684fd5b-qjmnh" event={"ID":"bef8682f-58cd-475a-a989-97a2a548fce0","Type":"ContainerStarted","Data":"f8e6f3e234f45d5dfe9e651c913032438905f0ed917400e88b371a7c953e992f"} Jul 17 05:45:48.095499 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.095469 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" event={"ID":"66ede3fa-03c0-4bfc-b099-865b673591ec","Type":"ContainerStarted","Data":"c4d1a56a89c30f38ce6ebc0ccf9947c5adcee2f951edd58c13bbecc424777484"} Jul 17 05:45:48.111903 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.111841 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-687684fd5b-qjmnh" podStartSLOduration=33.111820237 podStartE2EDuration="33.111820237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:48.111024938 +0000 UTC m=+275.300405221" watchObservedRunningTime="2026-07-17 05:45:48.111820237 +0000 UTC m=+275.301200524" Jul 17 05:45:48.692551 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.692517 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:48.695597 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:48.695572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:49.099969 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:49.099941 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:49.101756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:49.101728 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-687684fd5b-qjmnh" Jul 17 05:45:50.875907 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:50.875872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:50.878673 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:50.878640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63a926d0-adb4-453a-b7c1-568f38da49f2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cv9vc\" (UID: \"63a926d0-adb4-453a-b7c1-568f38da49f2\") " pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:51.049943 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.049910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cv9vc" Jul 17 05:45:51.110708 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.110235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" event={"ID":"66ede3fa-03c0-4bfc-b099-865b673591ec","Type":"ContainerStarted","Data":"08b7dc7093d36c9507acd6d9c319154804f8621fd81818160a3e02bd2b1211ad"} Jul 17 05:45:51.131637 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.131483 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-bhndn" podStartSLOduration=33.914543475 podStartE2EDuration="36.131462977s" totalImagesPullingTime="2.216919502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:51.129604823 +0000 UTC m=+278.318985106" watchObservedRunningTime="2026-07-17 05:45:51.131462977 +0000 UTC m=+278.320843261" Jul 17 05:45:51.198952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.198919 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cv9vc"] Jul 17 05:45:51.201860 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:51.201829 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a926d0_adb4_453a_b7c1_568f38da49f2.slice/crio-2ebc30ed9bf4bf2556bca7cf1c4b52e71f4a9873002b5cfc104b338bdeefba4c WatchSource:0}: Error finding container 2ebc30ed9bf4bf2556bca7cf1c4b52e71f4a9873002b5cfc104b338bdeefba4c: Status 404 returned error can't find the container with id 2ebc30ed9bf4bf2556bca7cf1c4b52e71f4a9873002b5cfc104b338bdeefba4c Jul 17 05:45:51.237109 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.237074 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:45:51.240194 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.240169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.242756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.242703 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 05:45:51.242756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.242706 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 05:45:51.243006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.242988 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 05:45:51.243215 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.243197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 05:45:51.243719 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.243699 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 05:45:51.244361 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.244334 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nglz8\"" Jul 17 05:45:51.255408 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.255386 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:45:51.379969 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.379939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.379969 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.379972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.380176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.380017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshgc\" (UniqueName: \"kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.380176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.380108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.380176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.380135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.380176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.380166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481765 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481765 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.481987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.481904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshgc\" (UniqueName: \"kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.482516 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.482467 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.482699 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.482677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.482889 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.482872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.484866 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.484844 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.484954 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.484918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.489502 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.489483 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshgc\" (UniqueName: \"kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc\") pod \"console-cb99bfc5f-hwdgx\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:51.551506 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:51.551468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:45:52.114828 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:52.114793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv9vc" event={"ID":"63a926d0-adb4-453a-b7c1-568f38da49f2","Type":"ContainerStarted","Data":"2ebc30ed9bf4bf2556bca7cf1c4b52e71f4a9873002b5cfc104b338bdeefba4c"} Jul 17 05:45:53.624518 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.624484 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2"] Jul 17 05:45:53.629889 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.629860 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.632220 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.632195 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 05:45:53.632383 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.632195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 05:45:53.633227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.633200 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 05:45:53.633227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.633222 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lkd6d\"" Jul 17 05:45:53.636132 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.636112 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2"] Jul 17 05:45:53.698647 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.698605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.698836 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.698666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.698836 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.698704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jn2\" (UniqueName: \"kubernetes.io/projected/d9712124-38f7-47d7-b035-9894209bac77-kube-api-access-j4jn2\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.698836 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.698777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9712124-38f7-47d7-b035-9894209bac77-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.799975 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.799937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9712124-38f7-47d7-b035-9894209bac77-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.800283 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.800051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.800283 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.800102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.800283 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.800140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jn2\" (UniqueName: \"kubernetes.io/projected/d9712124-38f7-47d7-b035-9894209bac77-kube-api-access-j4jn2\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.801348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.801301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9712124-38f7-47d7-b035-9894209bac77-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.804539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.804512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.804661 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.804619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d9712124-38f7-47d7-b035-9894209bac77-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.808605 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.808580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jn2\" (UniqueName: \"kubernetes.io/projected/d9712124-38f7-47d7-b035-9894209bac77-kube-api-access-j4jn2\") pod \"prometheus-operator-5945b5cfd-m4qm2\" (UID: \"d9712124-38f7-47d7-b035-9894209bac77\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:53.942306 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:53.942211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" Jul 17 05:45:54.785955 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:54.785910 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" Jul 17 05:45:54.785955 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:45:54.785910 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p2l9p" podUID="aa8fcf25-18bd-42b7-bc0b-bbd34870577a" Jul 17 05:45:55.124860 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:55.124829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:45:55.125031 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:55.124829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:45:58.499855 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.499557 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2"] Jul 17 05:45:58.501838 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:58.501806 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9712124_38f7_47d7_b035_9894209bac77.slice/crio-6c6b23af7e8c254fd25637438b7ef38f57185f206f69ccaa27b437128b921dc0 WatchSource:0}: Error finding container 6c6b23af7e8c254fd25637438b7ef38f57185f206f69ccaa27b437128b921dc0: Status 404 returned error can't find the container with id 6c6b23af7e8c254fd25637438b7ef38f57185f206f69ccaa27b437128b921dc0 Jul 17 05:45:58.527224 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.527197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:45:58.530641 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:45:58.530608 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8143021_cef9_4178_bd02_51b9a56b4b17.slice/crio-f92c6541e8c3bf3d2a16efd1a81fe3ed852768d12b4ee9af8f683658ec5d973d WatchSource:0}: Error finding container f92c6541e8c3bf3d2a16efd1a81fe3ed852768d12b4ee9af8f683658ec5d973d: Status 404 returned error can't find the container with id f92c6541e8c3bf3d2a16efd1a81fe3ed852768d12b4ee9af8f683658ec5d973d Jul 17 05:45:58.743480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.742871 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:45:58.743480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.742958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:45:58.746319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.746244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aa8fcf25-18bd-42b7-bc0b-bbd34870577a-metrics-tls\") pod \"dns-default-p2l9p\" (UID: \"aa8fcf25-18bd-42b7-bc0b-bbd34870577a\") " pod="openshift-dns/dns-default-p2l9p" Jul 17 05:45:58.748230 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:58.748203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"image-registry-7d99669ff8-c62jl\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:45:59.030927 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.030846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx7lr\"" Jul 17 05:45:59.031095 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.030846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hxf9n\"" Jul 17 05:45:59.037528 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.037430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:45:59.038206 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.037806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:45:59.152444 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.150863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wmplw" event={"ID":"b785f4f1-6406-434e-8376-e74896762cd5","Type":"ContainerStarted","Data":"d480b5bd4168e65f50856076edd974ed557accd01d423147ad95bc8c7b912572"} Jul 17 05:45:59.153646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.153619 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:59.155955 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.155832 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-wmplw" Jul 17 05:45:59.157915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.157628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" event={"ID":"d9712124-38f7-47d7-b035-9894209bac77","Type":"ContainerStarted","Data":"6c6b23af7e8c254fd25637438b7ef38f57185f206f69ccaa27b437128b921dc0"} Jul 17 05:45:59.160758 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.160702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb99bfc5f-hwdgx" event={"ID":"b8143021-cef9-4178-bd02-51b9a56b4b17","Type":"ContainerStarted","Data":"f92c6541e8c3bf3d2a16efd1a81fe3ed852768d12b4ee9af8f683658ec5d973d"} Jul 17 05:45:59.185166 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.184778 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-wmplw" podStartSLOduration=2.051556566 podStartE2EDuration="18.18468635s" totalImagesPullingTime="16.133129784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:45:59.179487067 +0000 UTC m=+286.368867350" watchObservedRunningTime="2026-07-17 05:45:59.18468635 +0000 UTC m=+286.374066651" Jul 17 05:45:59.275651 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.275618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p2l9p"] Jul 17 05:45:59.302105 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:45:59.302070 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:46:00.168222 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.168123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2l9p" event={"ID":"aa8fcf25-18bd-42b7-bc0b-bbd34870577a","Type":"ContainerStarted","Data":"87384870c412eeae1db9aaa0afbe82998e30aba316d82fde21cfd6bbcba04c8e"} Jul 17 05:46:00.172305 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.172154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" event={"ID":"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e","Type":"ContainerStarted","Data":"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec"} Jul 17 05:46:00.172305 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.172202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" event={"ID":"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e","Type":"ContainerStarted","Data":"20ad8bacea09eab024bbb9ba25b4cfe1bf3c76980cb3beaeb2e459501be27718"} Jul 17 05:46:00.172305 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.172273 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:46:00.195386 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.195292 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" podStartSLOduration=286.19527092 podStartE2EDuration="4m46.19527092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:00.194049424 +0000 UTC m=+287.383429708" watchObservedRunningTime="2026-07-17 05:46:00.19527092 +0000 UTC m=+287.384651205" Jul 17 05:46:00.828240 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.826966 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:00.862059 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.861165 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:00.862059 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.861316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.870480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.870243 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxpw8\" (UniqueName: \"kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:00.963936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:00.963820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.065474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.064949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.065474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.065474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.065474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.066796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.066796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.066796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.065631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxpw8\" (UniqueName: \"kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.067083 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.067058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.068125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.067320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.068125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.067715 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.068125 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.068075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.075857 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.075833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxpw8\" (UniqueName: \"kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.078626 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.078554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.079563 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.079523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert\") pod \"console-cccf45ff7-5485r\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.176597 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.176392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:01.490790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:01.490621 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:02.188230 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:02.187369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" event={"ID":"d9712124-38f7-47d7-b035-9894209bac77","Type":"ContainerStarted","Data":"673ec723bb27be46558d3ec590a019660d11c00696fa61d60bb28b1c557ec32f"} Jul 17 05:46:02.188230 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:02.187441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" event={"ID":"d9712124-38f7-47d7-b035-9894209bac77","Type":"ContainerStarted","Data":"c69ebb394857fe4c67c11b0ca25cfd9e57d70ed6b44d9b9dc0279fd85c31dab9"} Jul 17 05:46:02.191284 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:02.191176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cccf45ff7-5485r" event={"ID":"55d23d23-38b0-4e42-9ef9-3c67302cb222","Type":"ContainerStarted","Data":"4e265e9653691ce6e7dd2bc8a50067b1c1cc402ff8d8341d06fdb044efbf6325"} Jul 17 05:46:02.205988 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:02.205413 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m4qm2" podStartSLOduration=6.415139905 podStartE2EDuration="9.20538899s" totalImagesPullingTime="2.790249085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:02.204158502 +0000 UTC m=+289.393538812" watchObservedRunningTime="2026-07-17 05:46:02.20538899 +0000 UTC m=+289.394769274" Jul 17 05:46:03.987745 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:03.987706 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fzxfk"] Jul 17 05:46:04.000612 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.000585 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xzqcd"] Jul 17 05:46:04.000767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.000697 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.003396 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.003173 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzjkz\"" Jul 17 05:46:04.003396 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.003221 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 05:46:04.003396 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.003187 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 05:46:04.003656 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.003630 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 05:46:04.015480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.015441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xzqcd"] Jul 17 05:46:04.015598 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.015565 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.017908 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.017737 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 05:46:04.017908 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.017851 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 05:46:04.018095 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.017975 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4brkj\"" Jul 17 05:46:04.018231 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.018211 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 05:46:04.098740 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b603103-a2ad-440f-9f20-704f47915c0d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.098897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.098897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-sys\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.098897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099043 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.099043 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-textfile\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099043 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.098988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt67v\" (UniqueName: \"kubernetes.io/projected/da8479c5-33af-4dfb-856d-6605cf464a8a-kube-api-access-rt67v\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099169 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099169 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099169 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.099323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.099323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-root\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-metrics-client-ca\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099464 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-wtmp\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.099464 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.099391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8f7n\" (UniqueName: \"kubernetes.io/projected/2b603103-a2ad-440f-9f20-704f47915c0d-kube-api-access-m8f7n\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.199810 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.199810 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-textfile\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt67v\" (UniqueName: \"kubernetes.io/projected/da8479c5-33af-4dfb-856d-6605cf464a8a-kube-api-access-rt67v\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.199965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-root\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-textfile\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-root\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-metrics-client-ca\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-wtmp\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8f7n\" (UniqueName: \"kubernetes.io/projected/2b603103-a2ad-440f-9f20-704f47915c0d-kube-api-access-m8f7n\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b603103-a2ad-440f-9f20-704f47915c0d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-sys\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202036 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:04.201189 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-wtmp\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:04.201292 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls podName:2b603103-a2ad-440f-9f20-704f47915c0d nodeName:}" failed. No retries permitted until 2026-07-17 05:46:04.701230203 +0000 UTC m=+291.890610481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-xzqcd" (UID: "2b603103-a2ad-440f-9f20-704f47915c0d") : secret "kube-state-metrics-tls" not found Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:04.201320 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:04.201402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls podName:da8479c5-33af-4dfb-856d-6605cf464a8a nodeName:}" failed. No retries permitted until 2026-07-17 05:46:04.701386174 +0000 UTC m=+291.890766438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls") pod "node-exporter-fzxfk" (UID: "da8479c5-33af-4dfb-856d-6605cf464a8a") : secret "node-exporter-tls" not found Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.200835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b603103-a2ad-440f-9f20-704f47915c0d-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da8479c5-33af-4dfb-856d-6605cf464a8a-sys\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2b603103-a2ad-440f-9f20-704f47915c0d-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.202826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.201673 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.203210 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.203186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.205407 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.205364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.212163 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.212118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8f7n\" (UniqueName: \"kubernetes.io/projected/2b603103-a2ad-440f-9f20-704f47915c0d-kube-api-access-m8f7n\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.213513 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.213468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt67v\" (UniqueName: \"kubernetes.io/projected/da8479c5-33af-4dfb-856d-6605cf464a8a-kube-api-access-rt67v\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.219370 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.219321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da8479c5-33af-4dfb-856d-6605cf464a8a-metrics-client-ca\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.706442 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.706404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.706620 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.706474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.709498 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.709444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/da8479c5-33af-4dfb-856d-6605cf464a8a-node-exporter-tls\") pod \"node-exporter-fzxfk\" (UID: \"da8479c5-33af-4dfb-856d-6605cf464a8a\") " pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.714570 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.714524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b603103-a2ad-440f-9f20-704f47915c0d-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xzqcd\" (UID: \"2b603103-a2ad-440f-9f20-704f47915c0d\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:04.913606 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.913575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fzxfk" Jul 17 05:46:04.925736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:04.925675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" Jul 17 05:46:06.277868 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:06.277826 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-gr7tv" podUID="6617226d-ebd9-40ab-ae75-4fec0d433e35" Jul 17 05:46:08.359407 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:46:08.359365 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8479c5_33af_4dfb_856d_6605cf464a8a.slice/crio-42d599503f5b7c022adae22bf996e1111d4736a3bd06ca6f7414dfae16d08b8d WatchSource:0}: Error finding container 42d599503f5b7c022adae22bf996e1111d4736a3bd06ca6f7414dfae16d08b8d: Status 404 returned error can't find the container with id 42d599503f5b7c022adae22bf996e1111d4736a3bd06ca6f7414dfae16d08b8d Jul 17 05:46:08.519853 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:08.519783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xzqcd"] Jul 17 05:46:08.660156 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:46:08.660128 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b603103_a2ad_440f_9f20_704f47915c0d.slice/crio-320564387f89b93ffb97f16af270d8ecb5b4ebb12a0fd9a771be0cf071e92a2e WatchSource:0}: Error finding container 320564387f89b93ffb97f16af270d8ecb5b4ebb12a0fd9a771be0cf071e92a2e: Status 404 returned error can't find the container with id 320564387f89b93ffb97f16af270d8ecb5b4ebb12a0fd9a771be0cf071e92a2e Jul 17 05:46:09.022735 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.022701 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:46:09.048204 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.048174 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:46:09.064969 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.064881 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:46:09.065122 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.065007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.225787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.225750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzxfk" event={"ID":"da8479c5-33af-4dfb-856d-6605cf464a8a","Type":"ContainerStarted","Data":"42d599503f5b7c022adae22bf996e1111d4736a3bd06ca6f7414dfae16d08b8d"} Jul 17 05:46:09.227145 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.227116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cccf45ff7-5485r" event={"ID":"55d23d23-38b0-4e42-9ef9-3c67302cb222","Type":"ContainerStarted","Data":"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa"} Jul 17 05:46:09.228567 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.228544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv9vc" event={"ID":"63a926d0-adb4-453a-b7c1-568f38da49f2","Type":"ContainerStarted","Data":"461fe31f182aaa0d444c8b6b8cac21f56bbf3ea77918442e824eb5f70a484836"} Jul 17 05:46:09.230169 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.230149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2l9p" event={"ID":"aa8fcf25-18bd-42b7-bc0b-bbd34870577a","Type":"ContainerStarted","Data":"a89a65fe9e0de0ebee5cca508334032d2eb7ee40124ce00bed6ced0bf751357e"} Jul 17 05:46:09.230293 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.230184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p2l9p" event={"ID":"aa8fcf25-18bd-42b7-bc0b-bbd34870577a","Type":"ContainerStarted","Data":"4344902e3eaa08b30ecaa4989aec5c354c884bb93ffa06773ae5cc765fd35185"} Jul 17 05:46:09.230293 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.230213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:46:09.231634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.231615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb99bfc5f-hwdgx" event={"ID":"b8143021-cef9-4178-bd02-51b9a56b4b17","Type":"ContainerStarted","Data":"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5"} Jul 17 05:46:09.232671 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.232651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" event={"ID":"2b603103-a2ad-440f-9f20-704f47915c0d","Type":"ContainerStarted","Data":"320564387f89b93ffb97f16af270d8ecb5b4ebb12a0fd9a771be0cf071e92a2e"} Jul 17 05:46:09.244303 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.244263 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cccf45ff7-5485r" podStartSLOduration=2.306553023 podStartE2EDuration="9.244233212s" totalImagesPullingTime="6.937680189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:09.242315611 +0000 UTC m=+296.431695896" watchObservedRunningTime="2026-07-17 05:46:09.244233212 +0000 UTC m=+296.433613494" Jul 17 05:46:09.253476 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253560 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253497 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253560 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253629 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253629 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtc4l\" (UniqueName: \"kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253696 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.253729 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.253696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.258361 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.258324 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p2l9p" podStartSLOduration=252.245699874 podStartE2EDuration="4m21.258311102s" totalImagesPullingTime="9.012611228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:09.258300638 +0000 UTC m=+296.447680921" watchObservedRunningTime="2026-07-17 05:46:09.258311102 +0000 UTC m=+296.447691384" Jul 17 05:46:09.354268 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354447 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354447 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354561 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtc4l\" (UniqueName: \"kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354612 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354612 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.354773 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.354755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.355380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.355358 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.355502 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.355427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.356217 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.356180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.356525 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.356496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.358422 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.358400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.358679 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.358654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.364709 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.364687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtc4l\" (UniqueName: \"kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l\") pod \"console-758b48fddb-rhv7m\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.377441 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.377398 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:09.809443 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.809166 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cb99bfc5f-hwdgx" podStartSLOduration=8.986929653 podStartE2EDuration="18.809141582s" totalImagesPullingTime="9.822211929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:09.273382037 +0000 UTC m=+296.462762324" watchObservedRunningTime="2026-07-17 05:46:09.809141582 +0000 UTC m=+296.998521866" Jul 17 05:46:09.809443 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:09.809382 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:46:09.814192 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:46:09.814167 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8962f0c9_8ba4_409d_878d_1d395091f755.slice/crio-f22fcbeb6cd3e935f0616565563837122eadf0622074fd5e8becc2ec134f2b9e WatchSource:0}: Error finding container f22fcbeb6cd3e935f0616565563837122eadf0622074fd5e8becc2ec134f2b9e: Status 404 returned error can't find the container with id f22fcbeb6cd3e935f0616565563837122eadf0622074fd5e8becc2ec134f2b9e Jul 17 05:46:10.243239 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:10.243150 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b48fddb-rhv7m" event={"ID":"8962f0c9-8ba4-409d-878d-1d395091f755","Type":"ContainerStarted","Data":"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c"} Jul 17 05:46:10.243239 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:10.243238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b48fddb-rhv7m" event={"ID":"8962f0c9-8ba4-409d-878d-1d395091f755","Type":"ContainerStarted","Data":"f22fcbeb6cd3e935f0616565563837122eadf0622074fd5e8becc2ec134f2b9e"} Jul 17 05:46:10.266362 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:10.266217 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-758b48fddb-rhv7m" podStartSLOduration=1.266196736 podStartE2EDuration="1.266196736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:10.265703354 +0000 UTC m=+297.455083639" watchObservedRunningTime="2026-07-17 05:46:10.266196736 +0000 UTC m=+297.455577021" Jul 17 05:46:11.177370 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.177282 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:11.177370 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.177348 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:11.179344 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.179307 2574 patch_prober.go:28] interesting pod/console-cccf45ff7-5485r container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body= Jul 17 05:46:11.179476 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.179370 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-cccf45ff7-5485r" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerName="console" probeResult="failure" output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" Jul 17 05:46:11.248763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.248727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzxfk" event={"ID":"da8479c5-33af-4dfb-856d-6605cf464a8a","Type":"ContainerStarted","Data":"6488992c1a8439db9104d7e18c29d38ae7d8f60fd91161b19a5c068fa727d0b8"} Jul 17 05:46:11.551987 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:11.551909 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:46:13.255666 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:13.255632 2574 generic.go:356] "Generic (PLEG): container finished" podID="da8479c5-33af-4dfb-856d-6605cf464a8a" containerID="6488992c1a8439db9104d7e18c29d38ae7d8f60fd91161b19a5c068fa727d0b8" exitCode=0 Jul 17 05:46:13.255989 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:13.255700 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzxfk" event={"ID":"da8479c5-33af-4dfb-856d-6605cf464a8a","Type":"ContainerDied","Data":"6488992c1a8439db9104d7e18c29d38ae7d8f60fd91161b19a5c068fa727d0b8"} Jul 17 05:46:13.786410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:13.414171 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:46:13.786410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:13.414330 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:46:13.786410 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:13.417226 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 05:46:14.261417 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.261380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" event={"ID":"2b603103-a2ad-440f-9f20-704f47915c0d","Type":"ContainerStarted","Data":"55691633b2bcb0a880648dbed0c796ebc5d436274829644f0d73ff19f629309f"} Jul 17 05:46:14.261873 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.261448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" event={"ID":"2b603103-a2ad-440f-9f20-704f47915c0d","Type":"ContainerStarted","Data":"8c402a5b65f5471cd7f2f9b4e645bd78efb8ccda1d3652313642b01c3f643d6a"} Jul 17 05:46:14.261873 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.261478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" event={"ID":"2b603103-a2ad-440f-9f20-704f47915c0d","Type":"ContainerStarted","Data":"d748b4eb617427bb5505b9d30f2a5c504e8941af19cff31ca82bbfb2b2b40f81"} Jul 17 05:46:14.263963 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.263855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzxfk" event={"ID":"da8479c5-33af-4dfb-856d-6605cf464a8a","Type":"ContainerStarted","Data":"9f6cf6c1f55b0e8cf5603ca6320e8c717c360f6d2adc5e7e1626e06e748a0ff0"} Jul 17 05:46:14.264380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.264015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fzxfk" event={"ID":"da8479c5-33af-4dfb-856d-6605cf464a8a","Type":"ContainerStarted","Data":"bd6477eee503d87dbfa89bf000892d09120e3c28f70cccd89d7794c016bf77e6"} Jul 17 05:46:14.266390 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.266362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv9vc" event={"ID":"63a926d0-adb4-453a-b7c1-568f38da49f2","Type":"ContainerStarted","Data":"f747948a6a3eba64571f143be96872134c403b8ec517196a0e426cf9bc59e2e4"} Jul 17 05:46:14.277500 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.277445 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-xzqcd" podStartSLOduration=6.563525613 podStartE2EDuration="11.277427279s" totalImagesPullingTime="4.713901666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:14.276586801 +0000 UTC m=+301.465967097" watchObservedRunningTime="2026-07-17 05:46:14.277427279 +0000 UTC m=+301.466807563" Jul 17 05:46:14.293005 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:14.292960 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fzxfk" podStartSLOduration=8.928884961 podStartE2EDuration="11.292947154s" totalImagesPullingTime="2.264429661s" totalInitContainerRuntime="99.632532ms" isStatefulPod=true podCreationTimestamp="2026-07-17 05:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:14.292104042 +0000 UTC m=+301.481484326" watchObservedRunningTime="2026-07-17 05:46:14.292947154 +0000 UTC m=+301.482327437" Jul 17 05:46:19.042940 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.042908 2574 patch_prober.go:28] interesting pod/image-registry-7d99669ff8-c62jl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 05:46:19.043446 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.042967 2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 05:46:19.245672 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.245641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p2l9p" Jul 17 05:46:19.290879 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.290845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cv9vc" event={"ID":"63a926d0-adb4-453a-b7c1-568f38da49f2","Type":"ContainerStarted","Data":"ad9c0547b6f16b9ed62879e6f3a07a826337668514b16caf54e6c561e2098343"} Jul 17 05:46:19.309876 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.309781 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cv9vc" podStartSLOduration=34.449553734 podStartE2EDuration="1m1.309758812s" totalImagesPullingTime="26.860205078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 05:45:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:46:19.307636833 +0000 UTC m=+306.497017118" watchObservedRunningTime="2026-07-17 05:46:19.309758812 +0000 UTC m=+306.499139096" Jul 17 05:46:19.379157 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.378479 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:19.379157 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.379151 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:19.379672 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.379636 2574 patch_prober.go:28] interesting pod/console-758b48fddb-rhv7m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body= Jul 17 05:46:19.379805 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:19.379700 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-758b48fddb-rhv7m" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" containerName="console" probeResult="failure" output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" Jul 17 05:46:20.276123 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:20.276085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:46:21.180831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:21.180805 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:21.184547 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:21.184527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:22.195485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:22.195456 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:46:29.381549 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:29.381517 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:29.385205 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:29.385181 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:46:29.443223 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:29.443196 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:32.006394 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:32.006359 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:46:35.275073 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.275024 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cb99bfc5f-hwdgx" podUID="b8143021-cef9-4178-bd02-51b9a56b4b17" containerName="console" containerID="cri-o://e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5" gracePeriod=15 Jul 17 05:46:35.553830 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.553801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:46:35.698621 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698593 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zshgc\" (UniqueName: \"kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.698787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.698787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.698787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698729 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.698787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698760 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.698787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.698783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert\") pod \"b8143021-cef9-4178-bd02-51b9a56b4b17\" (UID: \"b8143021-cef9-4178-bd02-51b9a56b4b17\") " Jul 17 05:46:35.699148 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.699116 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:35.699320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.699162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config" (OuterVolumeSpecName: "console-config") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:35.699320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.699241 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:35.701146 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.701117 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:35.701319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.701114 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:35.701319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.701164 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc" (OuterVolumeSpecName: "kube-api-access-zshgc") pod "b8143021-cef9-4178-bd02-51b9a56b4b17" (UID: "b8143021-cef9-4178-bd02-51b9a56b4b17"). InnerVolumeSpecName "kube-api-access-zshgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799936 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799963 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799972 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799980 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8143021-cef9-4178-bd02-51b9a56b4b17-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799988 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8143021-cef9-4178-bd02-51b9a56b4b17-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:35.800015 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:35.799996 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshgc\" (UniqueName: \"kubernetes.io/projected/b8143021-cef9-4178-bd02-51b9a56b4b17-kube-api-access-zshgc\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:36.361088 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.361056 2574 generic.go:356] "Generic (PLEG): container finished" podID="b8143021-cef9-4178-bd02-51b9a56b4b17" containerID="e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5" exitCode=0 Jul 17 05:46:36.361472 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.361099 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cb99bfc5f-hwdgx" Jul 17 05:46:36.361472 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.361138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb99bfc5f-hwdgx" event={"ID":"b8143021-cef9-4178-bd02-51b9a56b4b17","Type":"ContainerDied","Data":"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5"} Jul 17 05:46:36.361472 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.361185 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cb99bfc5f-hwdgx" event={"ID":"b8143021-cef9-4178-bd02-51b9a56b4b17","Type":"ContainerDied","Data":"f92c6541e8c3bf3d2a16efd1a81fe3ed852768d12b4ee9af8f683658ec5d973d"} Jul 17 05:46:36.361472 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.361205 2574 scope.go:117] "RemoveContainer" containerID="e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5" Jul 17 05:46:36.370111 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.370092 2574 scope.go:117] "RemoveContainer" containerID="e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5" Jul 17 05:46:36.370420 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:36.370400 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5\": container with ID starting with e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5 not found: ID does not exist" containerID="e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5" Jul 17 05:46:36.370489 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.370429 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5"} err="failed to get container status \"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5\": rpc error: code = NotFound desc = could not find container \"e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5\": container with ID starting with e1678d4a02e79f78d7bdad834f3c67bbd4b279213d94fbc519f313f7e01947b5 not found: ID does not exist" Jul 17 05:46:36.382080 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.382061 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:46:36.387710 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:36.387689 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cb99bfc5f-hwdgx"] Jul 17 05:46:37.280932 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:37.280899 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8143021-cef9-4178-bd02-51b9a56b4b17" path="/var/lib/kubelet/pods/b8143021-cef9-4178-bd02-51b9a56b4b17/volumes" Jul 17 05:46:54.431219 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.431183 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcadf004-328f-4c25-bd51-34074c28ea9e" containerID="2485ff843784b6cbd039f3cace06c01e879cc2a4078411bb9bd5a110b6e05add" exitCode=0 Jul 17 05:46:54.431767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.431267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" event={"ID":"fcadf004-328f-4c25-bd51-34074c28ea9e","Type":"ContainerDied","Data":"2485ff843784b6cbd039f3cace06c01e879cc2a4078411bb9bd5a110b6e05add"} Jul 17 05:46:54.431767 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.431690 2574 scope.go:117] "RemoveContainer" containerID="2485ff843784b6cbd039f3cace06c01e879cc2a4078411bb9bd5a110b6e05add" Jul 17 05:46:54.472621 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.472579 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cccf45ff7-5485r" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerName="console" containerID="cri-o://0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa" gracePeriod=15 Jul 17 05:46:54.740997 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.740974 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:54.759460 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759624 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759477 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759624 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759512 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxpw8\" (UniqueName: \"kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759624 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759582 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759624 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759611 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759839 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759839 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759679 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config\") pod \"55d23d23-38b0-4e42-9ef9-3c67302cb222\" (UID: \"55d23d23-38b0-4e42-9ef9-3c67302cb222\") " Jul 17 05:46:54.759944 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759848 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config" (OuterVolumeSpecName: "console-config") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:54.759944 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759868 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:54.760045 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759943 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.760045 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.759962 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.760287 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.760265 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca" (OuterVolumeSpecName: "service-ca") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:54.760399 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.760369 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:54.762403 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.762375 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:54.762654 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.762626 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8" (OuterVolumeSpecName: "kube-api-access-mxpw8") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "kube-api-access-mxpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:46:54.763648 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.763617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "55d23d23-38b0-4e42-9ef9-3c67302cb222" (UID: "55d23d23-38b0-4e42-9ef9-3c67302cb222"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:54.860803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.860771 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxpw8\" (UniqueName: \"kubernetes.io/projected/55d23d23-38b0-4e42-9ef9-3c67302cb222-kube-api-access-mxpw8\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.860803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.860795 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.860803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.860804 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.860803 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.860813 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55d23d23-38b0-4e42-9ef9-3c67302cb222-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:54.861052 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:54.860821 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55d23d23-38b0-4e42-9ef9-3c67302cb222-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:55.436407 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.436359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-m5wkv" event={"ID":"fcadf004-328f-4c25-bd51-34074c28ea9e","Type":"ContainerStarted","Data":"932e4bb85deca09636ca78c5802704d0cae70a14c55a686501ebbd1441bc32ec"} Jul 17 05:46:55.437748 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.437720 2574 generic.go:356] "Generic (PLEG): container finished" podID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerID="0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa" exitCode=0 Jul 17 05:46:55.437888 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.437759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cccf45ff7-5485r" event={"ID":"55d23d23-38b0-4e42-9ef9-3c67302cb222","Type":"ContainerDied","Data":"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa"} Jul 17 05:46:55.437888 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.437778 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cccf45ff7-5485r" Jul 17 05:46:55.437888 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.437794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cccf45ff7-5485r" event={"ID":"55d23d23-38b0-4e42-9ef9-3c67302cb222","Type":"ContainerDied","Data":"4e265e9653691ce6e7dd2bc8a50067b1c1cc402ff8d8341d06fdb044efbf6325"} Jul 17 05:46:55.437888 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.437821 2574 scope.go:117] "RemoveContainer" containerID="0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa" Jul 17 05:46:55.443120 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.443102 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d99669ff8-c62jl_fd1f5f1d-a4bf-4acd-8594-ca17cb10750e/registry/0.log" Jul 17 05:46:55.446842 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.446827 2574 scope.go:117] "RemoveContainer" containerID="0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa" Jul 17 05:46:55.447107 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:55.447089 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa\": container with ID starting with 0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa not found: ID does not exist" containerID="0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa" Jul 17 05:46:55.447167 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.447117 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa"} err="failed to get container status \"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa\": rpc error: code = NotFound desc = could not find container \"0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa\": container with ID starting with 0206220d0fde348a81fb4924b1914b4903a1a019f67ceb660b719187743dd2fa not found: ID does not exist" Jul 17 05:46:55.466369 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.466344 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:55.472856 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.472829 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cccf45ff7-5485r"] Jul 17 05:46:55.833094 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:55.833061 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pft2v_7dbdb628-ab21-484c-9b77-17e81ddd6698/node-ca/0.log" Jul 17 05:46:56.232926 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:56.232837 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687684fd5b-qjmnh_bef8682f-58cd-475a-a989-97a2a548fce0/router/0.log" Jul 17 05:46:56.832673 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:56.832645 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tv449_d91c61b2-0c22-4fca-873b-f947862774c5/serve-healthcheck-canary/0.log" Jul 17 05:46:57.026890 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:57.026849 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerName="registry" containerID="cri-o://23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec" gracePeriod=30 Jul 17 05:46:57.281061 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:57.280985 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" path="/var/lib/kubelet/pods/55d23d23-38b0-4e42-9ef9-3c67302cb222/volumes" Jul 17 05:46:58.267934 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.267913 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:46:58.288672 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288639 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.288793 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.288793 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288739 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.288918 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288820 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.288918 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288856 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cljlz\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.288918 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288888 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.289066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288941 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.289066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.288965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets\") pod \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\" (UID: \"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e\") " Jul 17 05:46:58.289490 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.289457 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:58.290305 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.290277 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:46:58.292404 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.292369 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:46:58.292518 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.292417 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:46:58.293051 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.293018 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:58.294938 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.294913 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz" (OuterVolumeSpecName: "kube-api-access-cljlz") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "kube-api-access-cljlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:46:58.295110 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.295089 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:46:58.302975 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.302949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" (UID: "fd1f5f1d-a4bf-4acd-8594-ca17cb10750e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:46:58.390100 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390068 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-certificates\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390100 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390093 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-installation-pull-secrets\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390100 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390102 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-registry-tls\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390327 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390110 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-trusted-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390327 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390120 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-ca-trust-extracted\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390327 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390130 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-image-registry-private-configuration\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390327 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390138 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cljlz\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-kube-api-access-cljlz\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.390327 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.390146 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e-bound-sa-token\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:46:58.451943 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.451918 2574 generic.go:356] "Generic (PLEG): container finished" podID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerID="23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec" exitCode=0 Jul 17 05:46:58.452114 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.451950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" event={"ID":"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e","Type":"ContainerDied","Data":"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec"} Jul 17 05:46:58.452114 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.451968 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" Jul 17 05:46:58.452114 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.452000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d99669ff8-c62jl" event={"ID":"fd1f5f1d-a4bf-4acd-8594-ca17cb10750e","Type":"ContainerDied","Data":"20ad8bacea09eab024bbb9ba25b4cfe1bf3c76980cb3beaeb2e459501be27718"} Jul 17 05:46:58.452114 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.452022 2574 scope.go:117] "RemoveContainer" containerID="23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec" Jul 17 05:46:58.461719 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.461704 2574 scope.go:117] "RemoveContainer" containerID="23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec" Jul 17 05:46:58.461946 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:46:58.461931 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec\": container with ID starting with 23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec not found: ID does not exist" containerID="23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec" Jul 17 05:46:58.461984 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.461954 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec"} err="failed to get container status \"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec\": rpc error: code = NotFound desc = could not find container \"23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec\": container with ID starting with 23276c324329aa39aea134dc2ac61418ae50d401d047d157749a30519a983cec not found: ID does not exist" Jul 17 05:46:58.471683 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.471661 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:46:58.474773 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:58.474755 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d99669ff8-c62jl"] Jul 17 05:46:59.281409 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:46:59.281381 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" path="/var/lib/kubelet/pods/fd1f5f1d-a4bf-4acd-8594-ca17cb10750e/volumes" Jul 17 05:47:27.125480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.125442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:47:27.127896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.127872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6617226d-ebd9-40ab-ae75-4fec0d433e35-metrics-certs\") pod \"network-metrics-daemon-gr7tv\" (UID: \"6617226d-ebd9-40ab-ae75-4fec0d433e35\") " pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:47:27.179176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.179153 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7x5wz\"" Jul 17 05:47:27.188087 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.188058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gr7tv" Jul 17 05:47:27.317037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.317007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gr7tv"] Jul 17 05:47:27.320182 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:47:27.320151 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617226d_ebd9_40ab_ae75_4fec0d433e35.slice/crio-6f564935d8f2431c5d7063d76ef82d7cecfccbaadfdf859b1b265f3248f1f15c WatchSource:0}: Error finding container 6f564935d8f2431c5d7063d76ef82d7cecfccbaadfdf859b1b265f3248f1f15c: Status 404 returned error can't find the container with id 6f564935d8f2431c5d7063d76ef82d7cecfccbaadfdf859b1b265f3248f1f15c Jul 17 05:47:27.322214 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.322197 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:47:27.565026 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:27.564946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7tv" event={"ID":"6617226d-ebd9-40ab-ae75-4fec0d433e35","Type":"ContainerStarted","Data":"6f564935d8f2431c5d7063d76ef82d7cecfccbaadfdf859b1b265f3248f1f15c"} Jul 17 05:47:28.269436 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269390 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp"] Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269808 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerName="console" Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269826 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerName="console" Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269840 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerName="registry" Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269849 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerName="registry" Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269861 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8143021-cef9-4178-bd02-51b9a56b4b17" containerName="console" Jul 17 05:47:28.269936 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269870 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8143021-cef9-4178-bd02-51b9a56b4b17" containerName="console" Jul 17 05:47:28.270227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269946 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d23d23-38b0-4e42-9ef9-3c67302cb222" containerName="console" Jul 17 05:47:28.270227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269961 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1f5f1d-a4bf-4acd-8594-ca17cb10750e" containerName="registry" Jul 17 05:47:28.270227 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.269970 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8143021-cef9-4178-bd02-51b9a56b4b17" containerName="console" Jul 17 05:47:28.273991 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.273962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.276357 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276329 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 05:47:28.276492 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276385 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 05:47:28.276492 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276411 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 05:47:28.276492 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2nb22\"" Jul 17 05:47:28.276683 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276600 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 05:47:28.276683 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.276614 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 05:47:28.282521 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.281722 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 05:47:28.285088 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.285024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp"] Jul 17 05:47:28.437111 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437075 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-serving-certs-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2bb\" (UniqueName: \"kubernetes.io/projected/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-kube-api-access-sz2bb\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-metrics-client-ca\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437243 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-federate-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.437438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.437388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538679 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538679 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-federate-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.538897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-serving-certs-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.539059 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2bb\" (UniqueName: \"kubernetes.io/projected/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-kube-api-access-sz2bb\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.539059 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.538993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-metrics-client-ca\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.539717 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.539659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-serving-certs-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.539717 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.539705 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.540029 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.540003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-metrics-client-ca\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.541459 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.541435 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-telemeter-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.542035 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.542016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.542266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.542230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-secret-telemeter-client\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.542524 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.542509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-federate-client-tls\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.546096 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.546076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2bb\" (UniqueName: \"kubernetes.io/projected/9cb4570c-4dfd-4f95-9f6f-12e9fae27d79-kube-api-access-sz2bb\") pod \"telemeter-client-6bbd7d56d-jwqgp\" (UID: \"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79\") " pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.570755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.570727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7tv" event={"ID":"6617226d-ebd9-40ab-ae75-4fec0d433e35","Type":"ContainerStarted","Data":"2ddc2ce421515beef52ec1262941628d3825f700a195e2da764e9683f8d33a88"} Jul 17 05:47:28.570843 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.570786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gr7tv" event={"ID":"6617226d-ebd9-40ab-ae75-4fec0d433e35","Type":"ContainerStarted","Data":"84a57e451819e72c038df9714d525941295924ebf24e4aa7a3d45cadf3b265d3"} Jul 17 05:47:28.585928 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.585859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" Jul 17 05:47:28.589541 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.589492 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gr7tv" podStartSLOduration=373.5878838 podStartE2EDuration="6m14.589474061s" totalImagesPullingTime="1.001590261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:41:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:28.587868496 +0000 UTC m=+375.777248806" watchObservedRunningTime="2026-07-17 05:47:28.589474061 +0000 UTC m=+375.778854346" Jul 17 05:47:28.750092 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:28.750069 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp"] Jul 17 05:47:28.752313 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:47:28.752289 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb4570c_4dfd_4f95_9f6f_12e9fae27d79.slice/crio-fac7d687872b6dba550f66f8c4dabd44235fb7fb8714d58470d38c6bd7f38c7b WatchSource:0}: Error finding container fac7d687872b6dba550f66f8c4dabd44235fb7fb8714d58470d38c6bd7f38c7b: Status 404 returned error can't find the container with id fac7d687872b6dba550f66f8c4dabd44235fb7fb8714d58470d38c6bd7f38c7b Jul 17 05:47:29.575548 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:29.575514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" event={"ID":"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79","Type":"ContainerStarted","Data":"fac7d687872b6dba550f66f8c4dabd44235fb7fb8714d58470d38c6bd7f38c7b"} Jul 17 05:47:31.584397 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:31.584363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" event={"ID":"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79","Type":"ContainerStarted","Data":"330d88312b8df4ed8297debbaf2736265fd2ef77b84374e03895ab4bf80d3fb1"} Jul 17 05:47:32.590404 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:32.590372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" event={"ID":"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79","Type":"ContainerStarted","Data":"2738491f47f28d154ff91cae7d4028003e5f27229d52ea14c9ceeef72171d68e"} Jul 17 05:47:32.590805 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:32.590426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" event={"ID":"9cb4570c-4dfd-4f95-9f6f-12e9fae27d79","Type":"ContainerStarted","Data":"495136bc3d42e9b4c9771664795e0ba756472e0704c0120ac342b672ccd161ab"} Jul 17 05:47:32.614518 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:32.614448 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bbd7d56d-jwqgp" podStartSLOduration=1.8944367560000002 podStartE2EDuration="4.614428316s" totalImagesPullingTime="2.71999156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:47:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:32.613077546 +0000 UTC m=+379.802457841" watchObservedRunningTime="2026-07-17 05:47:32.614428316 +0000 UTC m=+379.803808599" Jul 17 05:47:33.355939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.355899 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:47:33.358591 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.358562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.371025 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.370959 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:47:33.477600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477783 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7sr\" (UniqueName: \"kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.477981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.477876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.578862 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7sr\" (UniqueName: \"kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579078 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.578992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.579717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.579737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.579769 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.579767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.580092 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.580060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.583365 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.583341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.583508 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.583486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.586377 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.586359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7sr\" (UniqueName: \"kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr\") pod \"console-764595dffd-wm5nb\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.668242 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.668150 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:33.813073 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:33.813038 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:47:33.815945 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:47:33.815915 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694edca9_ab2b_421c_bd22_2cc896efcbf5.slice/crio-21c35787504c479f25dfd922ed86a973ce89fa6306400cbe7254dd24986a48c0 WatchSource:0}: Error finding container 21c35787504c479f25dfd922ed86a973ce89fa6306400cbe7254dd24986a48c0: Status 404 returned error can't find the container with id 21c35787504c479f25dfd922ed86a973ce89fa6306400cbe7254dd24986a48c0 Jul 17 05:47:34.600870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:34.600833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764595dffd-wm5nb" event={"ID":"694edca9-ab2b-421c-bd22-2cc896efcbf5","Type":"ContainerStarted","Data":"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10"} Jul 17 05:47:34.601023 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:34.600881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764595dffd-wm5nb" event={"ID":"694edca9-ab2b-421c-bd22-2cc896efcbf5","Type":"ContainerStarted","Data":"21c35787504c479f25dfd922ed86a973ce89fa6306400cbe7254dd24986a48c0"} Jul 17 05:47:34.619311 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:34.619263 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-764595dffd-wm5nb" podStartSLOduration=1.6192287699999999 podStartE2EDuration="1.61922877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:47:34.618021048 +0000 UTC m=+381.807401331" watchObservedRunningTime="2026-07-17 05:47:34.61922877 +0000 UTC m=+381.808609055" Jul 17 05:47:43.668731 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:43.668699 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:43.669224 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:43.668760 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:43.673525 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:43.673507 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:44.643089 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:44.643061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:47:44.689509 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:47:44.689479 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:48:09.715057 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:09.715013 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-758b48fddb-rhv7m" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" containerName="console" containerID="cri-o://d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c" gracePeriod=15 Jul 17 05:48:09.975552 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:09.975499 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:48:10.059387 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059358 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtc4l\" (UniqueName: \"kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059551 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.059800 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059578 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config\") pod \"8962f0c9-8ba4-409d-878d-1d395091f755\" (UID: \"8962f0c9-8ba4-409d-878d-1d395091f755\") " Jul 17 05:48:10.060079 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca" (OuterVolumeSpecName: "service-ca") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:48:10.060079 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.059963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:48:10.060079 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.060030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:48:10.060079 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.060044 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config" (OuterVolumeSpecName: "console-config") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:48:10.061701 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.061675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:48:10.061866 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.061846 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l" (OuterVolumeSpecName: "kube-api-access-vtc4l") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "kube-api-access-vtc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:48:10.061958 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.061936 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8962f0c9-8ba4-409d-878d-1d395091f755" (UID: "8962f0c9-8ba4-409d-878d-1d395091f755"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:48:10.160787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160756 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.160787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160782 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.160787 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160790 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.161006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160800 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.161006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160809 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8962f0c9-8ba4-409d-878d-1d395091f755-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.161006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160818 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8962f0c9-8ba4-409d-878d-1d395091f755-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.161006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.160826 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtc4l\" (UniqueName: \"kubernetes.io/projected/8962f0c9-8ba4-409d-878d-1d395091f755-kube-api-access-vtc4l\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:48:10.736062 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.736032 2574 generic.go:356] "Generic (PLEG): container finished" podID="8962f0c9-8ba4-409d-878d-1d395091f755" containerID="d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c" exitCode=0 Jul 17 05:48:10.736526 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.736086 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-758b48fddb-rhv7m" Jul 17 05:48:10.736526 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.736096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b48fddb-rhv7m" event={"ID":"8962f0c9-8ba4-409d-878d-1d395091f755","Type":"ContainerDied","Data":"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c"} Jul 17 05:48:10.736526 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.736138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-758b48fddb-rhv7m" event={"ID":"8962f0c9-8ba4-409d-878d-1d395091f755","Type":"ContainerDied","Data":"f22fcbeb6cd3e935f0616565563837122eadf0622074fd5e8becc2ec134f2b9e"} Jul 17 05:48:10.736526 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.736157 2574 scope.go:117] "RemoveContainer" containerID="d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c" Jul 17 05:48:10.744785 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.744766 2574 scope.go:117] "RemoveContainer" containerID="d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c" Jul 17 05:48:10.745049 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:48:10.745031 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c\": container with ID starting with d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c not found: ID does not exist" containerID="d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c" Jul 17 05:48:10.745096 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.745057 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c"} err="failed to get container status \"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c\": rpc error: code = NotFound desc = could not find container \"d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c\": container with ID starting with d7e3f524d9e194d8ea5f3bd2296cf3f41beda116193f79656bcc7cf22f545b8c not found: ID does not exist" Jul 17 05:48:10.757189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.757165 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:48:10.762437 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:10.762412 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-758b48fddb-rhv7m"] Jul 17 05:48:11.281213 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:11.281181 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" path="/var/lib/kubelet/pods/8962f0c9-8ba4-409d-878d-1d395091f755/volumes" Jul 17 05:48:42.122552 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.122521 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:48:42.123034 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.122798 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" containerName="console" Jul 17 05:48:42.123034 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.122808 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" containerName="console" Jul 17 05:48:42.123034 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.122864 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8962f0c9-8ba4-409d-878d-1d395091f755" containerName="console" Jul 17 05:48:42.125468 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.125446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.136317 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.136296 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:48:42.188634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.188634 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.188870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.188870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw2g6\" (UniqueName: \"kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.188870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.188870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.189023 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.188871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289592 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289592 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw2g6\" (UniqueName: \"kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289763 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.289944 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.289765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.290444 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.290421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.290564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.290463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.290784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.290762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.290897 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.290873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.292242 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.292219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.292375 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.292357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.296513 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.296495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw2g6\" (UniqueName: \"kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6\") pod \"console-64774445fb-sp257\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.434438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.434379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:42.570365 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.570336 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:48:42.572891 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:48:42.572865 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6a41d2_865b_4eee_be71_a475ac38ab60.slice/crio-81e8ca89ab1fbb1af7fb29fe5a60ee02d3439ffbe40c6eced092535124e18b1e WatchSource:0}: Error finding container 81e8ca89ab1fbb1af7fb29fe5a60ee02d3439ffbe40c6eced092535124e18b1e: Status 404 returned error can't find the container with id 81e8ca89ab1fbb1af7fb29fe5a60ee02d3439ffbe40c6eced092535124e18b1e Jul 17 05:48:42.856271 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.856221 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64774445fb-sp257" event={"ID":"5b6a41d2-865b-4eee-be71-a475ac38ab60","Type":"ContainerStarted","Data":"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc"} Jul 17 05:48:42.856432 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.856304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64774445fb-sp257" event={"ID":"5b6a41d2-865b-4eee-be71-a475ac38ab60","Type":"ContainerStarted","Data":"81e8ca89ab1fbb1af7fb29fe5a60ee02d3439ffbe40c6eced092535124e18b1e"} Jul 17 05:48:42.872576 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:42.872536 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64774445fb-sp257" podStartSLOduration=0.872521603 podStartE2EDuration="872.521603ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:48:42.872089354 +0000 UTC m=+450.061469635" watchObservedRunningTime="2026-07-17 05:48:42.872521603 +0000 UTC m=+450.061901885" Jul 17 05:48:52.434495 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:52.434447 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:52.434905 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:52.434518 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:52.439438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:52.439418 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:52.897139 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:52.897110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:48:52.936802 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:48:52.936770 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:49:17.963995 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:17.963950 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-764595dffd-wm5nb" podUID="694edca9-ab2b-421c-bd22-2cc896efcbf5" containerName="console" containerID="cri-o://e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10" gracePeriod=15 Jul 17 05:49:18.207144 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.207120 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:49:18.377164 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377137 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377177 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377195 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377218 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377284 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377306 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p7sr\" (UniqueName: \"kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config\") pod \"694edca9-ab2b-421c-bd22-2cc896efcbf5\" (UID: \"694edca9-ab2b-421c-bd22-2cc896efcbf5\") " Jul 17 05:49:18.377791 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377747 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca" (OuterVolumeSpecName: "service-ca") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:49:18.377791 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377755 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:49:18.377948 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config" (OuterVolumeSpecName: "console-config") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:49:18.377948 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.377850 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:49:18.379517 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.379496 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:49:18.379620 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.379578 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:49:18.379701 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.379684 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr" (OuterVolumeSpecName: "kube-api-access-2p7sr") pod "694edca9-ab2b-421c-bd22-2cc896efcbf5" (UID: "694edca9-ab2b-421c-bd22-2cc896efcbf5"). InnerVolumeSpecName "kube-api-access-2p7sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:49:18.478733 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478710 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478733 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478730 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478739 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478747 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478756 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478764 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p7sr\" (UniqueName: \"kubernetes.io/projected/694edca9-ab2b-421c-bd22-2cc896efcbf5-kube-api-access-2p7sr\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.478864 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.478773 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/694edca9-ab2b-421c-bd22-2cc896efcbf5-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:18.995489 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.995460 2574 generic.go:356] "Generic (PLEG): container finished" podID="694edca9-ab2b-421c-bd22-2cc896efcbf5" containerID="e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10" exitCode=0 Jul 17 05:49:18.995909 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.995509 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764595dffd-wm5nb" Jul 17 05:49:18.995909 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.995531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764595dffd-wm5nb" event={"ID":"694edca9-ab2b-421c-bd22-2cc896efcbf5","Type":"ContainerDied","Data":"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10"} Jul 17 05:49:18.995909 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.995571 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764595dffd-wm5nb" event={"ID":"694edca9-ab2b-421c-bd22-2cc896efcbf5","Type":"ContainerDied","Data":"21c35787504c479f25dfd922ed86a973ce89fa6306400cbe7254dd24986a48c0"} Jul 17 05:49:18.995909 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:18.995590 2574 scope.go:117] "RemoveContainer" containerID="e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10" Jul 17 05:49:19.004543 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:19.004527 2574 scope.go:117] "RemoveContainer" containerID="e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10" Jul 17 05:49:19.004797 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:49:19.004777 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10\": container with ID starting with e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10 not found: ID does not exist" containerID="e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10" Jul 17 05:49:19.004842 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:19.004807 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10"} err="failed to get container status \"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10\": rpc error: code = NotFound desc = could not find container \"e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10\": container with ID starting with e19148b835afacceee6395c0de871e2f144256806e62f615c0dc4ba6f8052a10 not found: ID does not exist" Jul 17 05:49:19.014375 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:19.014351 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:49:19.017554 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:19.017536 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-764595dffd-wm5nb"] Jul 17 05:49:19.281209 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:19.281139 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="694edca9-ab2b-421c-bd22-2cc896efcbf5" path="/var/lib/kubelet/pods/694edca9-ab2b-421c-bd22-2cc896efcbf5/volumes" Jul 17 05:49:21.258916 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.258885 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg"] Jul 17 05:49:21.259328 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.259217 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694edca9-ab2b-421c-bd22-2cc896efcbf5" containerName="console" Jul 17 05:49:21.259328 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.259228 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="694edca9-ab2b-421c-bd22-2cc896efcbf5" containerName="console" Jul 17 05:49:21.259328 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.259298 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="694edca9-ab2b-421c-bd22-2cc896efcbf5" containerName="console" Jul 17 05:49:21.263826 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.263810 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.266365 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.266344 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 05:49:21.266474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.266349 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 05:49:21.266474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.266368 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4h7vg\"" Jul 17 05:49:21.270095 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.270072 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg"] Jul 17 05:49:21.299201 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.299172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568sk\" (UniqueName: \"kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.299347 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.299212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.299347 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.299306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.399766 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.399732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.399927 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.399782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568sk\" (UniqueName: \"kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.399927 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.399823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.400188 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.400166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.400290 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.400176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.408234 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.408213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-568sk\" (UniqueName: \"kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.573922 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.573852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:21.703775 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:21.703752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg"] Jul 17 05:49:21.705553 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:49:21.705521 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9609df42_e647_4276_ab3b_24a72c2b58d2.slice/crio-0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5 WatchSource:0}: Error finding container 0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5: Status 404 returned error can't find the container with id 0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5 Jul 17 05:49:22.008325 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:22.008293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" event={"ID":"9609df42-e647-4276-ab3b-24a72c2b58d2","Type":"ContainerStarted","Data":"0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5"} Jul 17 05:49:28.037060 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:28.037026 2574 generic.go:356] "Generic (PLEG): container finished" podID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerID="c14ef36ab4102351873562d4715269855eb00b4a4ef37100070253186b1121ef" exitCode=0 Jul 17 05:49:28.037540 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:28.037065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" event={"ID":"9609df42-e647-4276-ab3b-24a72c2b58d2","Type":"ContainerDied","Data":"c14ef36ab4102351873562d4715269855eb00b4a4ef37100070253186b1121ef"} Jul 17 05:49:31.050635 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:31.050604 2574 generic.go:356] "Generic (PLEG): container finished" podID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerID="16c283d5874c326f09f0be42779bfec8374b42f7f05f05fab3818854044b5201" exitCode=0 Jul 17 05:49:31.051066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:31.050685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" event={"ID":"9609df42-e647-4276-ab3b-24a72c2b58d2","Type":"ContainerDied","Data":"16c283d5874c326f09f0be42779bfec8374b42f7f05f05fab3818854044b5201"} Jul 17 05:49:38.084103 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:38.084064 2574 generic.go:356] "Generic (PLEG): container finished" podID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerID="49bd919ede58a2c358afc127afc962fec1487e9e1ad0256d6c82033e4e20fede" exitCode=0 Jul 17 05:49:38.084505 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:38.084167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" event={"ID":"9609df42-e647-4276-ab3b-24a72c2b58d2","Type":"ContainerDied","Data":"49bd919ede58a2c358afc127afc962fec1487e9e1ad0256d6c82033e4e20fede"} Jul 17 05:49:39.228449 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.228417 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:39.354361 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.354290 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-568sk\" (UniqueName: \"kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk\") pod \"9609df42-e647-4276-ab3b-24a72c2b58d2\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " Jul 17 05:49:39.354507 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.354367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle\") pod \"9609df42-e647-4276-ab3b-24a72c2b58d2\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " Jul 17 05:49:39.354507 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.354392 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util\") pod \"9609df42-e647-4276-ab3b-24a72c2b58d2\" (UID: \"9609df42-e647-4276-ab3b-24a72c2b58d2\") " Jul 17 05:49:39.355003 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.354981 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle" (OuterVolumeSpecName: "bundle") pod "9609df42-e647-4276-ab3b-24a72c2b58d2" (UID: "9609df42-e647-4276-ab3b-24a72c2b58d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:49:39.356698 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.356655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk" (OuterVolumeSpecName: "kube-api-access-568sk") pod "9609df42-e647-4276-ab3b-24a72c2b58d2" (UID: "9609df42-e647-4276-ab3b-24a72c2b58d2"). InnerVolumeSpecName "kube-api-access-568sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:49:39.359312 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.359290 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util" (OuterVolumeSpecName: "util") pod "9609df42-e647-4276-ab3b-24a72c2b58d2" (UID: "9609df42-e647-4276-ab3b-24a72c2b58d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:49:39.455373 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.455345 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568sk\" (UniqueName: \"kubernetes.io/projected/9609df42-e647-4276-ab3b-24a72c2b58d2-kube-api-access-568sk\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:39.455373 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.455369 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:39.455373 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:39.455379 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9609df42-e647-4276-ab3b-24a72c2b58d2-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:40.093098 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:40.093064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" event={"ID":"9609df42-e647-4276-ab3b-24a72c2b58d2","Type":"ContainerDied","Data":"0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5"} Jul 17 05:49:40.093098 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:40.093096 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540k8gsg" Jul 17 05:49:40.093329 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:40.093113 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b94a18e7892be7f373a16b67263303b056dc877bcef2feb2f1155a9ba03b8c5" Jul 17 05:49:44.545295 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545241 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt"] Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545663 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="pull" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545681 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="pull" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545701 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="util" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545708 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="util" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545731 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="extract" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545740 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="extract" Jul 17 05:49:44.545863 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.545816 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9609df42-e647-4276-ab3b-24a72c2b58d2" containerName="extract" Jul 17 05:49:44.548894 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.548874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.551378 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.551356 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 17 05:49:44.551489 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.551384 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mg74h\"" Jul 17 05:49:44.551489 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.551359 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:49:44.561961 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.561942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt"] Jul 17 05:49:44.700181 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.700148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx5xn\" (UniqueName: \"kubernetes.io/projected/9628877c-088e-4960-9935-8e189e48996c-kube-api-access-xx5xn\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.700367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.700208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9628877c-088e-4960-9935-8e189e48996c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.801314 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.801210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9628877c-088e-4960-9935-8e189e48996c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.801475 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.801324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx5xn\" (UniqueName: \"kubernetes.io/projected/9628877c-088e-4960-9935-8e189e48996c-kube-api-access-xx5xn\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.802116 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.802092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9628877c-088e-4960-9935-8e189e48996c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.809113 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.809090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx5xn\" (UniqueName: \"kubernetes.io/projected/9628877c-088e-4960-9935-8e189e48996c-kube-api-access-xx5xn\") pod \"cert-manager-operator-controller-manager-55c444bf86-zj2qt\" (UID: \"9628877c-088e-4960-9935-8e189e48996c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.858070 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.858047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" Jul 17 05:49:44.989490 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:44.989460 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt"] Jul 17 05:49:44.993034 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:49:44.993009 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9628877c_088e_4960_9935_8e189e48996c.slice/crio-ba0735448ef9ea4bda3228db7491c80ad3ce1d781814fef5498ead18546fc230 WatchSource:0}: Error finding container ba0735448ef9ea4bda3228db7491c80ad3ce1d781814fef5498ead18546fc230: Status 404 returned error can't find the container with id ba0735448ef9ea4bda3228db7491c80ad3ce1d781814fef5498ead18546fc230 Jul 17 05:49:45.114424 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:45.114395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" event={"ID":"9628877c-088e-4960-9935-8e189e48996c","Type":"ContainerStarted","Data":"ba0735448ef9ea4bda3228db7491c80ad3ce1d781814fef5498ead18546fc230"} Jul 17 05:49:47.124647 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:47.124609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" event={"ID":"9628877c-088e-4960-9935-8e189e48996c","Type":"ContainerStarted","Data":"2478b927b5c2240231daab2c1d7a870dfc7927dacc011ad201e2f7bca837c8aa"} Jul 17 05:49:47.147381 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:47.147330 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zj2qt" podStartSLOduration=1.454619991 podStartE2EDuration="3.147313317s" totalImagesPullingTime="1.692693326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:49:47.145834549 +0000 UTC m=+514.335214833" watchObservedRunningTime="2026-07-17 05:49:47.147313317 +0000 UTC m=+514.336693626" Jul 17 05:49:48.479230 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.479197 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx"] Jul 17 05:49:48.481548 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.481528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.484136 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.484115 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 05:49:48.484136 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.484127 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4h7vg\"" Jul 17 05:49:48.484939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.484923 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 05:49:48.490462 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.490440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx"] Jul 17 05:49:48.632368 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.632335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.632368 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.632371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.632583 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.632410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx57g\" (UniqueName: \"kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.733479 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.733396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.733479 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.733434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.733479 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.733473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx57g\" (UniqueName: \"kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.733781 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.733762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.733824 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.733794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.741398 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.741370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx57g\" (UniqueName: \"kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.790916 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.790892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:48.918488 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:48.918322 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx"] Jul 17 05:49:48.920504 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:49:48.920468 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a75a78_d889_480a_81e6_9732c6ad3b72.slice/crio-34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc WatchSource:0}: Error finding container 34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc: Status 404 returned error can't find the container with id 34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc Jul 17 05:49:49.133721 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:49.133684 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerID="6c9420bd060c78a4347055b7bb46c229dd15d4555d21287a0c8acad299489955" exitCode=0 Jul 17 05:49:49.133885 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:49.133739 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" event={"ID":"a8a75a78-d889-480a-81e6-9732c6ad3b72","Type":"ContainerDied","Data":"6c9420bd060c78a4347055b7bb46c229dd15d4555d21287a0c8acad299489955"} Jul 17 05:49:49.133885 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:49.133789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" event={"ID":"a8a75a78-d889-480a-81e6-9732c6ad3b72","Type":"ContainerStarted","Data":"34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc"} Jul 17 05:49:51.150262 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.150212 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerID="c65580ea33cd1a27cc8b1207538e4f0f998473d4ff8bd42b9e6473dd6d2bb061" exitCode=0 Jul 17 05:49:51.150609 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.150277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" event={"ID":"a8a75a78-d889-480a-81e6-9732c6ad3b72","Type":"ContainerDied","Data":"c65580ea33cd1a27cc8b1207538e4f0f998473d4ff8bd42b9e6473dd6d2bb061"} Jul 17 05:49:51.307323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.307294 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn"] Jul 17 05:49:51.309387 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.309374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.311870 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.311846 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 05:49:51.311961 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.311877 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 05:49:51.311961 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.311855 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-97d5v\"" Jul 17 05:49:51.317434 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.317413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn"] Jul 17 05:49:51.458703 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.458677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-kube-api-access-rk4p6\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.458839 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.458714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.560108 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.560037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-kube-api-access-rk4p6\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.560108 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.560071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.567812 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.567780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.567995 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.567973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/eac3439f-ede0-4083-b9a3-fdf23bea02f1-kube-api-access-rk4p6\") pod \"cert-manager-cainjector-559cdd8c87-5pmpn\" (UID: \"eac3439f-ede0-4083-b9a3-fdf23bea02f1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.650317 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.650293 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" Jul 17 05:49:51.784438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:51.784283 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn"] Jul 17 05:49:51.786562 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:49:51.786535 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac3439f_ede0_4083_b9a3_fdf23bea02f1.slice/crio-e75204fd31f9b4df179fe3cd4ddb4ed28bb9035f0cfc9ba0dda9f6c68bfef959 WatchSource:0}: Error finding container e75204fd31f9b4df179fe3cd4ddb4ed28bb9035f0cfc9ba0dda9f6c68bfef959: Status 404 returned error can't find the container with id e75204fd31f9b4df179fe3cd4ddb4ed28bb9035f0cfc9ba0dda9f6c68bfef959 Jul 17 05:49:52.156383 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:52.156352 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerID="9ba447f3bb71c3d040aedb339e2adde8997be0c3f8dd316118546c6a8ae39389" exitCode=0 Jul 17 05:49:52.156822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:52.156439 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" event={"ID":"a8a75a78-d889-480a-81e6-9732c6ad3b72","Type":"ContainerDied","Data":"9ba447f3bb71c3d040aedb339e2adde8997be0c3f8dd316118546c6a8ae39389"} Jul 17 05:49:52.157609 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:52.157587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" event={"ID":"eac3439f-ede0-4083-b9a3-fdf23bea02f1","Type":"ContainerStarted","Data":"e75204fd31f9b4df179fe3cd4ddb4ed28bb9035f0cfc9ba0dda9f6c68bfef959"} Jul 17 05:49:54.150948 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.150927 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:54.169493 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.169462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" Jul 17 05:49:54.169638 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.169461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftp5qx" event={"ID":"a8a75a78-d889-480a-81e6-9732c6ad3b72","Type":"ContainerDied","Data":"34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc"} Jul 17 05:49:54.169638 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.169596 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ef216125f82200ec57cf5dcdde842f9280e7c34e792e1717ac3219322390dc" Jul 17 05:49:54.285922 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.285899 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util\") pod \"a8a75a78-d889-480a-81e6-9732c6ad3b72\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " Jul 17 05:49:54.286048 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.285934 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle\") pod \"a8a75a78-d889-480a-81e6-9732c6ad3b72\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " Jul 17 05:49:54.286048 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.286021 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx57g\" (UniqueName: \"kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g\") pod \"a8a75a78-d889-480a-81e6-9732c6ad3b72\" (UID: \"a8a75a78-d889-480a-81e6-9732c6ad3b72\") " Jul 17 05:49:54.286340 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.286315 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle" (OuterVolumeSpecName: "bundle") pod "a8a75a78-d889-480a-81e6-9732c6ad3b72" (UID: "a8a75a78-d889-480a-81e6-9732c6ad3b72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:49:54.288178 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.288153 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g" (OuterVolumeSpecName: "kube-api-access-vx57g") pod "a8a75a78-d889-480a-81e6-9732c6ad3b72" (UID: "a8a75a78-d889-480a-81e6-9732c6ad3b72"). InnerVolumeSpecName "kube-api-access-vx57g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:49:54.290275 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.290236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util" (OuterVolumeSpecName: "util") pod "a8a75a78-d889-480a-81e6-9732c6ad3b72" (UID: "a8a75a78-d889-480a-81e6-9732c6ad3b72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:49:54.387066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.386990 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx57g\" (UniqueName: \"kubernetes.io/projected/a8a75a78-d889-480a-81e6-9732c6ad3b72-kube-api-access-vx57g\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:54.387066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.387016 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:54.387066 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:54.387025 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8a75a78-d889-480a-81e6-9732c6ad3b72-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:49:55.175461 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:55.175424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" event={"ID":"eac3439f-ede0-4083-b9a3-fdf23bea02f1","Type":"ContainerStarted","Data":"71fefe337a6f564b1f47ed29c739d93646955102a1ea0045e57567fc60704306"} Jul 17 05:49:55.190699 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:49:55.190645 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5pmpn" podStartSLOduration=1.777092236 podStartE2EDuration="4.190629679s" totalImagesPullingTime="2.413537443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:49:55.189400965 +0000 UTC m=+522.378781274" watchObservedRunningTime="2026-07-17 05:49:55.190629679 +0000 UTC m=+522.380009962" Jul 17 05:50:01.083851 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.083822 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt"] Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084115 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="util" Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084125 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="util" Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084143 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="pull" Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084148 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="pull" Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084154 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="extract" Jul 17 05:50:01.084193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084159 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="extract" Jul 17 05:50:01.084463 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.084208 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a75a78-d889-480a-81e6-9732c6ad3b72" containerName="extract" Jul 17 05:50:01.087100 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.087084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.089474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.089452 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 05:50:01.089561 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.089501 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 05:50:01.090735 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.090716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-669sd\"" Jul 17 05:50:01.095452 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.095431 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt"] Jul 17 05:50:01.241855 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.241826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3123a727-91f7-48de-9e24-fd8fd1166153-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.242063 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.241882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/3123a727-91f7-48de-9e24-fd8fd1166153-kube-api-access-2dvrl\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.342930 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.342856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3123a727-91f7-48de-9e24-fd8fd1166153-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.342930 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.342913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/3123a727-91f7-48de-9e24-fd8fd1166153-kube-api-access-2dvrl\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.343207 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.343189 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3123a727-91f7-48de-9e24-fd8fd1166153-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.351320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.351294 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvrl\" (UniqueName: \"kubernetes.io/projected/3123a727-91f7-48de-9e24-fd8fd1166153-kube-api-access-2dvrl\") pod \"openshift-lws-operator-fd8ccff4c-d6xdt\" (UID: \"3123a727-91f7-48de-9e24-fd8fd1166153\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.396999 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.396975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" Jul 17 05:50:01.403645 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.403621 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mx9nd"] Jul 17 05:50:01.407016 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.406998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.409082 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.409066 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4gkxg\"" Jul 17 05:50:01.419176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.419154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mx9nd"] Jul 17 05:50:01.535382 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.535357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt"] Jul 17 05:50:01.537890 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:01.537851 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3123a727_91f7_48de_9e24_fd8fd1166153.slice/crio-f822e6d3b3dc2d8db419c77d4277316218a653d755d9a3691ec7397c13b92839 WatchSource:0}: Error finding container f822e6d3b3dc2d8db419c77d4277316218a653d755d9a3691ec7397c13b92839: Status 404 returned error can't find the container with id f822e6d3b3dc2d8db419c77d4277316218a653d755d9a3691ec7397c13b92839 Jul 17 05:50:01.544670 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.544639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-bound-sa-token\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.544832 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.544694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vq8q\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-kube-api-access-4vq8q\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.646179 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.646089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-bound-sa-token\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.646179 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.646134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vq8q\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-kube-api-access-4vq8q\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.654309 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.654286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-bound-sa-token\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.654430 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.654405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vq8q\" (UniqueName: \"kubernetes.io/projected/6f9c0d42-4d66-4960-b575-fac4a8da3102-kube-api-access-4vq8q\") pod \"cert-manager-55b4d954d5-mx9nd\" (UID: \"6f9c0d42-4d66-4960-b575-fac4a8da3102\") " pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.729671 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.729643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mx9nd" Jul 17 05:50:01.857983 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:01.857959 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mx9nd"] Jul 17 05:50:01.860425 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:01.860400 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f9c0d42_4d66_4960_b575_fac4a8da3102.slice/crio-e85d9917b04586cf179f14009408b0908205f1c34b7f93e76f71e7000fe9c743 WatchSource:0}: Error finding container e85d9917b04586cf179f14009408b0908205f1c34b7f93e76f71e7000fe9c743: Status 404 returned error can't find the container with id e85d9917b04586cf179f14009408b0908205f1c34b7f93e76f71e7000fe9c743 Jul 17 05:50:02.209555 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:02.209464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mx9nd" event={"ID":"6f9c0d42-4d66-4960-b575-fac4a8da3102","Type":"ContainerStarted","Data":"7eac68e90ec7a6eb60d42f74c03a52f98121b2a92ef9383369fd467c7db63585"} Jul 17 05:50:02.209555 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:02.209510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mx9nd" event={"ID":"6f9c0d42-4d66-4960-b575-fac4a8da3102","Type":"ContainerStarted","Data":"e85d9917b04586cf179f14009408b0908205f1c34b7f93e76f71e7000fe9c743"} Jul 17 05:50:02.210872 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:02.210842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" event={"ID":"3123a727-91f7-48de-9e24-fd8fd1166153","Type":"ContainerStarted","Data":"f822e6d3b3dc2d8db419c77d4277316218a653d755d9a3691ec7397c13b92839"} Jul 17 05:50:02.223605 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:02.223556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mx9nd" podStartSLOduration=1.223542581 podStartE2EDuration="1.223542581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:02.223054972 +0000 UTC m=+529.412435254" watchObservedRunningTime="2026-07-17 05:50:02.223542581 +0000 UTC m=+529.412922863" Jul 17 05:50:04.222265 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:04.222207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" event={"ID":"3123a727-91f7-48de-9e24-fd8fd1166153","Type":"ContainerStarted","Data":"75dc2964111b93b7e1c7067837d3faff2625ede6a0e226981296db73fdeefed0"} Jul 17 05:50:04.237603 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:04.237546 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d6xdt" podStartSLOduration=1.084881547 podStartE2EDuration="3.237525479s" totalImagesPullingTime="2.152643932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:04.235809895 +0000 UTC m=+531.425190197" watchObservedRunningTime="2026-07-17 05:50:04.237525479 +0000 UTC m=+531.426905763" Jul 17 05:50:07.340625 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.340585 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9"] Jul 17 05:50:07.344717 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.344694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.347176 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.347155 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 05:50:07.348233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.348212 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4h7vg\"" Jul 17 05:50:07.348358 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.348313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 05:50:07.350071 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.350048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9"] Jul 17 05:50:07.496831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.496796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vrb\" (UniqueName: \"kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.496831 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.496834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.497034 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.496874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.597952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.597873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vrb\" (UniqueName: \"kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.597952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.597906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.597952 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.597931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.598271 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.598235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.598326 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.598282 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.606095 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.606057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vrb\" (UniqueName: \"kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.655128 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.655095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:07.786914 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:07.786882 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9"] Jul 17 05:50:07.789833 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:07.789804 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b781a61_306a_4d3b_92de_8492739cf222.slice/crio-8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad WatchSource:0}: Error finding container 8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad: Status 404 returned error can't find the container with id 8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad Jul 17 05:50:08.242383 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:08.242300 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b781a61-306a-4d3b-92de-8492739cf222" containerID="b0ee77318bfc7e13db4a3a9fff6cbec6cea8ca11c54ce459af215b8fa63b20cb" exitCode=0 Jul 17 05:50:08.242757 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:08.242389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" event={"ID":"2b781a61-306a-4d3b-92de-8492739cf222","Type":"ContainerDied","Data":"b0ee77318bfc7e13db4a3a9fff6cbec6cea8ca11c54ce459af215b8fa63b20cb"} Jul 17 05:50:08.242757 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:08.242447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" event={"ID":"2b781a61-306a-4d3b-92de-8492739cf222","Type":"ContainerStarted","Data":"8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad"} Jul 17 05:50:10.254671 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:10.254634 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b781a61-306a-4d3b-92de-8492739cf222" containerID="dc5fe7be2e4ae94d510322bb810135e6a4b2c1e627baffe161aa946a7ac49916" exitCode=0 Jul 17 05:50:10.255060 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:10.254705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" event={"ID":"2b781a61-306a-4d3b-92de-8492739cf222","Type":"ContainerDied","Data":"dc5fe7be2e4ae94d510322bb810135e6a4b2c1e627baffe161aa946a7ac49916"} Jul 17 05:50:11.261092 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:11.261061 2574 generic.go:356] "Generic (PLEG): container finished" podID="2b781a61-306a-4d3b-92de-8492739cf222" containerID="e8e589bba65510526b4883c5a2b09efc790d1fc5714c49d69a9803bee212d3da" exitCode=0 Jul 17 05:50:11.261503 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:11.261137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" event={"ID":"2b781a61-306a-4d3b-92de-8492739cf222","Type":"ContainerDied","Data":"e8e589bba65510526b4883c5a2b09efc790d1fc5714c49d69a9803bee212d3da"} Jul 17 05:50:12.406033 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.406009 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:12.434509 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.434485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle\") pod \"2b781a61-306a-4d3b-92de-8492739cf222\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " Jul 17 05:50:12.434632 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.434524 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util\") pod \"2b781a61-306a-4d3b-92de-8492739cf222\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " Jul 17 05:50:12.434632 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.434553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6vrb\" (UniqueName: \"kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb\") pod \"2b781a61-306a-4d3b-92de-8492739cf222\" (UID: \"2b781a61-306a-4d3b-92de-8492739cf222\") " Jul 17 05:50:12.435274 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.435221 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle" (OuterVolumeSpecName: "bundle") pod "2b781a61-306a-4d3b-92de-8492739cf222" (UID: "2b781a61-306a-4d3b-92de-8492739cf222"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:50:12.436990 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.436958 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb" (OuterVolumeSpecName: "kube-api-access-n6vrb") pod "2b781a61-306a-4d3b-92de-8492739cf222" (UID: "2b781a61-306a-4d3b-92de-8492739cf222"). InnerVolumeSpecName "kube-api-access-n6vrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:50:12.441318 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.441292 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util" (OuterVolumeSpecName: "util") pod "2b781a61-306a-4d3b-92de-8492739cf222" (UID: "2b781a61-306a-4d3b-92de-8492739cf222"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:50:12.535856 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.535795 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:50:12.535856 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.535816 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b781a61-306a-4d3b-92de-8492739cf222-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:50:12.535856 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:12.535825 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6vrb\" (UniqueName: \"kubernetes.io/projected/2b781a61-306a-4d3b-92de-8492739cf222-kube-api-access-n6vrb\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:50:13.271268 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:13.271227 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" Jul 17 05:50:13.271392 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:13.271228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft4bx9" event={"ID":"2b781a61-306a-4d3b-92de-8492739cf222","Type":"ContainerDied","Data":"8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad"} Jul 17 05:50:13.271392 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:13.271387 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8399c36bbffd19d9399fbaf1059f50acea290b233db8bc9e417224ec9217c9ad" Jul 17 05:50:25.018170 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018136 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl"] Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018521 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="util" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018538 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="util" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="pull" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018563 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="pull" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="extract" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018577 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="extract" Jul 17 05:50:25.018697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.018635 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b781a61-306a-4d3b-92de-8492739cf222" containerName="extract" Jul 17 05:50:25.021630 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.021612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.024333 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.024307 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 05:50:25.024474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.024310 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xnd5q\"" Jul 17 05:50:25.024474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.024339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 05:50:25.024600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.024476 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 05:50:25.024600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.024586 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 05:50:25.040631 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.040608 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl"] Jul 17 05:50:25.136801 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.136772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvt5\" (UniqueName: \"kubernetes.io/projected/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-kube-api-access-mwvt5\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.136955 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.136809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.136955 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.136834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.136955 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.136895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.136955 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.136942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238344 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvt5\" (UniqueName: \"kubernetes.io/projected/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-kube-api-access-mwvt5\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238531 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.238899 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.238875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-manifests\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.240868 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.240847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-tmp\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.241011 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.240994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.241164 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.241144 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.247199 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.247176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvt5\" (UniqueName: \"kubernetes.io/projected/80784fe4-77e9-4f12-92fe-06bbfbd6f1f2-kube-api-access-mwvt5\") pod \"opendatahub-operator-controller-manager-554df894d5-mzhrl\" (UID: \"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2\") " pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.331996 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.331970 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:25.468785 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:25.468754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl"] Jul 17 05:50:25.472357 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:25.472324 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80784fe4_77e9_4f12_92fe_06bbfbd6f1f2.slice/crio-224a1a493bc96802afa0575bdbaca4b18782c36a4905b51626aa20c6529352ba WatchSource:0}: Error finding container 224a1a493bc96802afa0575bdbaca4b18782c36a4905b51626aa20c6529352ba: Status 404 returned error can't find the container with id 224a1a493bc96802afa0575bdbaca4b18782c36a4905b51626aa20c6529352ba Jul 17 05:50:26.331900 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:26.331861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" event={"ID":"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2","Type":"ContainerStarted","Data":"224a1a493bc96802afa0575bdbaca4b18782c36a4905b51626aa20c6529352ba"} Jul 17 05:50:29.348337 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:29.348308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" event={"ID":"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2","Type":"ContainerStarted","Data":"44371ebc8a9f6f7f0bc4638345a719a8b8ec0fd818fed3788d9349bc74970abd"} Jul 17 05:50:30.353850 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:30.353813 2574 generic.go:356] "Generic (PLEG): container finished" podID="80784fe4-77e9-4f12-92fe-06bbfbd6f1f2" containerID="44371ebc8a9f6f7f0bc4638345a719a8b8ec0fd818fed3788d9349bc74970abd" exitCode=0 Jul 17 05:50:30.354232 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:30.353884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" event={"ID":"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2","Type":"ContainerDied","Data":"44371ebc8a9f6f7f0bc4638345a719a8b8ec0fd818fed3788d9349bc74970abd"} Jul 17 05:50:31.359938 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:31.359897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" event={"ID":"80784fe4-77e9-4f12-92fe-06bbfbd6f1f2","Type":"ContainerStarted","Data":"ae86faed1144e3cd3a5a96a21ff20727b74b794d59388748cd5f95b2baa2682d"} Jul 17 05:50:31.360447 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:31.359991 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:31.382495 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:31.382444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" podStartSLOduration=3.16662383 podStartE2EDuration="7.382428901s" totalImagesPullingTime="3.201136426s" totalInitContainerRuntime="1.014668645s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:31.379724503 +0000 UTC m=+558.569104796" watchObservedRunningTime="2026-07-17 05:50:31.382428901 +0000 UTC m=+558.571809243" Jul 17 05:50:42.366541 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:42.366512 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-554df894d5-mzhrl" Jul 17 05:50:50.076627 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.076593 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p"] Jul 17 05:50:50.081324 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.081304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.083682 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.083651 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 05:50:50.083832 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.083702 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 05:50:50.083832 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.083712 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tnrsj\"" Jul 17 05:50:50.091183 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.091158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p"] Jul 17 05:50:50.248604 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.248558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvm7r\" (UniqueName: \"kubernetes.io/projected/bc1dbc69-7342-4d58-ae94-4d84acdf7350-kube-api-access-tvm7r\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.248829 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.248607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tls-certs\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.248829 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.248637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tmp\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.350111 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.350066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvm7r\" (UniqueName: \"kubernetes.io/projected/bc1dbc69-7342-4d58-ae94-4d84acdf7350-kube-api-access-tvm7r\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.350307 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.350124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tls-certs\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.350307 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.350154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tmp\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.352600 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.352575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tmp\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.353465 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.353438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1dbc69-7342-4d58-ae94-4d84acdf7350-tls-certs\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.361680 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.361654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvm7r\" (UniqueName: \"kubernetes.io/projected/bc1dbc69-7342-4d58-ae94-4d84acdf7350-kube-api-access-tvm7r\") pod \"kube-auth-proxy-79d55f4645-p5w2p\" (UID: \"bc1dbc69-7342-4d58-ae94-4d84acdf7350\") " pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.392471 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.392441 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" Jul 17 05:50:50.532662 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:50.532608 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p"] Jul 17 05:50:50.534370 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:50.534342 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1dbc69_7342_4d58_ae94_4d84acdf7350.slice/crio-fce59600c701144997c1c3fc428e86a496c9cac0ab52506136a742dcaf446715 WatchSource:0}: Error finding container fce59600c701144997c1c3fc428e86a496c9cac0ab52506136a742dcaf446715: Status 404 returned error can't find the container with id fce59600c701144997c1c3fc428e86a496c9cac0ab52506136a742dcaf446715 Jul 17 05:50:51.450964 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:51.450928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" event={"ID":"bc1dbc69-7342-4d58-ae94-4d84acdf7350","Type":"ContainerStarted","Data":"fce59600c701144997c1c3fc428e86a496c9cac0ab52506136a742dcaf446715"} Jul 17 05:50:52.163478 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.163444 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg"] Jul 17 05:50:52.167112 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.167086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.169937 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.169912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 05:50:52.170974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.170881 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xsjtl\"" Jul 17 05:50:52.170974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.170915 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 05:50:52.170974 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.170936 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 05:50:52.176770 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.176737 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg"] Jul 17 05:50:52.266584 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.266546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.266790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.266594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-manager-config\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.266790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.266719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hv5\" (UniqueName: \"kubernetes.io/projected/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-kube-api-access-97hv5\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.266790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.266755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.367758 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.367724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hv5\" (UniqueName: \"kubernetes.io/projected/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-kube-api-access-97hv5\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.367758 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.367758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.367998 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.367891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.367998 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.367971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-manager-config\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.368624 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.368542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-manager-config\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.370428 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.370408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-metrics-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.370564 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.370544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-cert\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.378748 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.378725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hv5\" (UniqueName: \"kubernetes.io/projected/81aa3bc8-ab3e-4b17-b958-bf8e174bc01d-kube-api-access-97hv5\") pod \"lws-controller-manager-d9bdd98d9-lmdvg\" (UID: \"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d\") " pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:52.481429 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:52.481346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:53.230119 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:53.230057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg"] Jul 17 05:50:53.231041 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:50:53.231016 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aa3bc8_ab3e_4b17_b958_bf8e174bc01d.slice/crio-390c698e54c99d3318da9c33f982db3d6935350c5c47c9c6eb205d0ccf90c901 WatchSource:0}: Error finding container 390c698e54c99d3318da9c33f982db3d6935350c5c47c9c6eb205d0ccf90c901: Status 404 returned error can't find the container with id 390c698e54c99d3318da9c33f982db3d6935350c5c47c9c6eb205d0ccf90c901 Jul 17 05:50:53.462452 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:53.462364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" event={"ID":"bc1dbc69-7342-4d58-ae94-4d84acdf7350","Type":"ContainerStarted","Data":"5de7c4a2a65898a35eb2cd403ed617f349d53ce50ddced43be3a809fcb37ca56"} Jul 17 05:50:53.463549 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:53.463523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" event={"ID":"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d","Type":"ContainerStarted","Data":"390c698e54c99d3318da9c33f982db3d6935350c5c47c9c6eb205d0ccf90c901"} Jul 17 05:50:53.484692 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:53.484645 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79d55f4645-p5w2p" podStartSLOduration=0.882960216 podStartE2EDuration="3.484630551s" totalImagesPullingTime="2.601670335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:53.483504036 +0000 UTC m=+580.672884335" watchObservedRunningTime="2026-07-17 05:50:53.484630551 +0000 UTC m=+580.674010833" Jul 17 05:50:55.478915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:55.478881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" event={"ID":"81aa3bc8-ab3e-4b17-b958-bf8e174bc01d","Type":"ContainerStarted","Data":"53eb473bb15831294869f9a2701b5c4eec3b959d179e17925626d5026466e288"} Jul 17 05:50:55.479383 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:55.479018 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:50:55.498916 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:50:55.498868 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" podStartSLOduration=2.238271357 podStartE2EDuration="3.498852705s" totalImagesPullingTime="1.260581348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:50:55.497449347 +0000 UTC m=+582.686829632" watchObservedRunningTime="2026-07-17 05:50:55.498852705 +0000 UTC m=+582.688232987" Jul 17 05:51:06.485833 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:51:06.485796 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d9bdd98d9-lmdvg" Jul 17 05:51:13.443308 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:51:13.443184 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:51:13.443308 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:51:13.443184 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:52:00.778894 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.778858 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk"] Jul 17 05:52:00.782405 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.782389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:00.784740 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.784719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 05:52:00.785471 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.785451 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cksw5\"" Jul 17 05:52:00.785548 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.785480 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 05:52:00.790023 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.790007 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk"] Jul 17 05:52:00.946400 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.946360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:00.946571 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.946405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsq2\" (UniqueName: \"kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:00.946571 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:00.946522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.047158 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.047089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.047158 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.047122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsq2\" (UniqueName: \"kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.047339 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.047172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.047477 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.047460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.047520 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.047481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.055211 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.055188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsq2\" (UniqueName: \"kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.092908 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.092883 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:01.224019 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.223984 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk"] Jul 17 05:52:01.227374 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:52:01.227344 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52050454_892f_4b64_b86e_416d3efdf46e.slice/crio-500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52 WatchSource:0}: Error finding container 500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52: Status 404 returned error can't find the container with id 500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52 Jul 17 05:52:01.579127 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.579091 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf"] Jul 17 05:52:01.582526 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.582509 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.590323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.590036 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf"] Jul 17 05:52:01.753885 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.753843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkv86\" (UniqueName: \"kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.753885 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.753890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.754127 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.753912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.785377 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.785341 2574 generic.go:356] "Generic (PLEG): container finished" podID="52050454-892f-4b64-b86e-416d3efdf46e" containerID="92337e09c25408f3365e0669e9d495bd567b890801a585af564eeb125321be2e" exitCode=0 Jul 17 05:52:01.785819 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.785426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" event={"ID":"52050454-892f-4b64-b86e-416d3efdf46e","Type":"ContainerDied","Data":"92337e09c25408f3365e0669e9d495bd567b890801a585af564eeb125321be2e"} Jul 17 05:52:01.785819 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.785473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" event={"ID":"52050454-892f-4b64-b86e-416d3efdf46e","Type":"ContainerStarted","Data":"500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52"} Jul 17 05:52:01.854588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.854487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.854588 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.854535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.854822 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.854622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkv86\" (UniqueName: \"kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.854982 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.854959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.855060 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.854964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.862730 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.862709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkv86\" (UniqueName: \"kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.892534 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.892506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:01.970844 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.970809 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5"] Jul 17 05:52:01.974529 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.974510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:01.987171 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:01.987146 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5"] Jul 17 05:52:02.035094 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.035070 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf"] Jul 17 05:52:02.036844 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:52:02.036821 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb628d4d_7904_4fdc_91e5_2fec37975821.slice/crio-65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011 WatchSource:0}: Error finding container 65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011: Status 404 returned error can't find the container with id 65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011 Jul 17 05:52:02.157362 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.157336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kf4\" (UniqueName: \"kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.157485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.157375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.157485 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.157400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.258490 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.258457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kf4\" (UniqueName: \"kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.258697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.258501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.258697 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.258526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.258862 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.258842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.258913 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.258872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.266541 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.266516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kf4\" (UniqueName: \"kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.287521 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.287495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:02.370945 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.370912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl"] Jul 17 05:52:02.375347 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.375326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.381540 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.381404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl"] Jul 17 05:52:02.439360 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.439329 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5"] Jul 17 05:52:02.439548 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:52:02.439524 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2f9e11_811e_41c7_b3dd_b85dc31f0425.slice/crio-8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47 WatchSource:0}: Error finding container 8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47: Status 404 returned error can't find the container with id 8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47 Jul 17 05:52:02.561678 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.561642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.561825 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.561707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klqtd\" (UniqueName: \"kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.561913 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.561883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.663236 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.663200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.663433 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.663269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.663433 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.663303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klqtd\" (UniqueName: \"kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.663616 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.663595 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.663683 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.663638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.671694 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.671674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klqtd\" (UniqueName: \"kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.689480 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.689424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:02.792635 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.792604 2574 generic.go:356] "Generic (PLEG): container finished" podID="db628d4d-7904-4fdc-91e5-2fec37975821" containerID="31a1eabda26b6ea69ff283a028719c641509bf8c6f314897648d1e89620217b2" exitCode=0 Jul 17 05:52:02.793117 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.792688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" event={"ID":"db628d4d-7904-4fdc-91e5-2fec37975821","Type":"ContainerDied","Data":"31a1eabda26b6ea69ff283a028719c641509bf8c6f314897648d1e89620217b2"} Jul 17 05:52:02.793117 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.792780 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" event={"ID":"db628d4d-7904-4fdc-91e5-2fec37975821","Type":"ContainerStarted","Data":"65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011"} Jul 17 05:52:02.794685 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.794660 2574 generic.go:356] "Generic (PLEG): container finished" podID="52050454-892f-4b64-b86e-416d3efdf46e" containerID="9e3c83c7cd0150de27fb4e39a152e6b9a63aa4c3e5046ae4ea9729d6bc13b658" exitCode=0 Jul 17 05:52:02.794820 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.794751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" event={"ID":"52050454-892f-4b64-b86e-416d3efdf46e","Type":"ContainerDied","Data":"9e3c83c7cd0150de27fb4e39a152e6b9a63aa4c3e5046ae4ea9729d6bc13b658"} Jul 17 05:52:02.800013 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.799992 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerID="c1c2e988f96ae26f7714b9af603406739652ac4f4ef4441ef41eb779d5473872" exitCode=0 Jul 17 05:52:02.800128 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.800021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" event={"ID":"5c2f9e11-811e-41c7-b3dd-b85dc31f0425","Type":"ContainerDied","Data":"c1c2e988f96ae26f7714b9af603406739652ac4f4ef4441ef41eb779d5473872"} Jul 17 05:52:02.800128 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.800054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" event={"ID":"5c2f9e11-811e-41c7-b3dd-b85dc31f0425","Type":"ContainerStarted","Data":"8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47"} Jul 17 05:52:02.858302 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:02.858277 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl"] Jul 17 05:52:02.859827 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:52:02.859804 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a58965_19eb_4337_9b87_f4c81ecc92b4.slice/crio-db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964 WatchSource:0}: Error finding container db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964: Status 404 returned error can't find the container with id db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964 Jul 17 05:52:03.806929 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.806898 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerID="72d88b41cc3062e57826bde09374013c9c09955603680cca0ac17373c1d8fb6f" exitCode=0 Jul 17 05:52:03.807388 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.806968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" event={"ID":"5c2f9e11-811e-41c7-b3dd-b85dc31f0425","Type":"ContainerDied","Data":"72d88b41cc3062e57826bde09374013c9c09955603680cca0ac17373c1d8fb6f"} Jul 17 05:52:03.808929 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.808906 2574 generic.go:356] "Generic (PLEG): container finished" podID="db628d4d-7904-4fdc-91e5-2fec37975821" containerID="991816e55519096b7bb1cc1449afc219fa606a8ee73ad0d9253373a081b8cb32" exitCode=0 Jul 17 05:52:03.809061 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.808942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" event={"ID":"db628d4d-7904-4fdc-91e5-2fec37975821","Type":"ContainerDied","Data":"991816e55519096b7bb1cc1449afc219fa606a8ee73ad0d9253373a081b8cb32"} Jul 17 05:52:03.811238 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.811220 2574 generic.go:356] "Generic (PLEG): container finished" podID="52050454-892f-4b64-b86e-416d3efdf46e" containerID="b87edfdc2c60c9759a4bb53c91dcbc21490ae292955539e719d2c9c293b7961a" exitCode=0 Jul 17 05:52:03.811336 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.811280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" event={"ID":"52050454-892f-4b64-b86e-416d3efdf46e","Type":"ContainerDied","Data":"b87edfdc2c60c9759a4bb53c91dcbc21490ae292955539e719d2c9c293b7961a"} Jul 17 05:52:03.812939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.812923 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerID="83db1b4e643ceb4310dccf6ed29527c8444cc641eca9af725ff1bb7d18f85720" exitCode=0 Jul 17 05:52:03.813018 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.812983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" event={"ID":"e4a58965-19eb-4337-9b87-f4c81ecc92b4","Type":"ContainerDied","Data":"83db1b4e643ceb4310dccf6ed29527c8444cc641eca9af725ff1bb7d18f85720"} Jul 17 05:52:03.813018 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:03.813006 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" event={"ID":"e4a58965-19eb-4337-9b87-f4c81ecc92b4","Type":"ContainerStarted","Data":"db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964"} Jul 17 05:52:04.826089 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.826063 2574 generic.go:356] "Generic (PLEG): container finished" podID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerID="a4b08813a353bcd34b44ebaf641d6bc658cdfa730af2537bc02e4d0be2368965" exitCode=0 Jul 17 05:52:04.826456 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.826145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" event={"ID":"5c2f9e11-811e-41c7-b3dd-b85dc31f0425","Type":"ContainerDied","Data":"a4b08813a353bcd34b44ebaf641d6bc658cdfa730af2537bc02e4d0be2368965"} Jul 17 05:52:04.828394 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.828370 2574 generic.go:356] "Generic (PLEG): container finished" podID="db628d4d-7904-4fdc-91e5-2fec37975821" containerID="94e6e9b37231dbf287395e5c9cb919196d15c485e2fa0e178cc93dd5fbc0e012" exitCode=0 Jul 17 05:52:04.828510 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.828451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" event={"ID":"db628d4d-7904-4fdc-91e5-2fec37975821","Type":"ContainerDied","Data":"94e6e9b37231dbf287395e5c9cb919196d15c485e2fa0e178cc93dd5fbc0e012"} Jul 17 05:52:04.967275 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.967234 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:04.979545 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.979525 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util\") pod \"52050454-892f-4b64-b86e-416d3efdf46e\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " Jul 17 05:52:04.979619 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.979553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvsq2\" (UniqueName: \"kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2\") pod \"52050454-892f-4b64-b86e-416d3efdf46e\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " Jul 17 05:52:04.979619 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.979594 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle\") pod \"52050454-892f-4b64-b86e-416d3efdf46e\" (UID: \"52050454-892f-4b64-b86e-416d3efdf46e\") " Jul 17 05:52:04.980206 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.980177 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle" (OuterVolumeSpecName: "bundle") pod "52050454-892f-4b64-b86e-416d3efdf46e" (UID: "52050454-892f-4b64-b86e-416d3efdf46e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:04.981876 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.981836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2" (OuterVolumeSpecName: "kube-api-access-cvsq2") pod "52050454-892f-4b64-b86e-416d3efdf46e" (UID: "52050454-892f-4b64-b86e-416d3efdf46e"). InnerVolumeSpecName "kube-api-access-cvsq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:52:04.985898 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:04.985874 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util" (OuterVolumeSpecName: "util") pod "52050454-892f-4b64-b86e-416d3efdf46e" (UID: "52050454-892f-4b64-b86e-416d3efdf46e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:05.080196 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.080167 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:05.080319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.080199 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsq2\" (UniqueName: \"kubernetes.io/projected/52050454-892f-4b64-b86e-416d3efdf46e-kube-api-access-cvsq2\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:05.080319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.080214 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52050454-892f-4b64-b86e-416d3efdf46e-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:05.837957 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.837930 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" Jul 17 05:52:05.838423 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.837926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk" event={"ID":"52050454-892f-4b64-b86e-416d3efdf46e","Type":"ContainerDied","Data":"500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52"} Jul 17 05:52:05.838423 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.838056 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500d08763351434de5e903a4577f92daed918fe0a3aaf3981a49af5499c4be52" Jul 17 05:52:05.839896 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.839860 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerID="9d6c4f61491793ecc43c0b61fb975a45cd7f9d9dc12eb4b9d9c2b25eb1ce5904" exitCode=0 Jul 17 05:52:05.840050 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:05.839961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" event={"ID":"e4a58965-19eb-4337-9b87-f4c81ecc92b4","Type":"ContainerDied","Data":"9d6c4f61491793ecc43c0b61fb975a45cd7f9d9dc12eb4b9d9c2b25eb1ce5904"} Jul 17 05:52:06.000457 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.000434 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:06.061363 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.061333 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:06.086649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086621 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle\") pod \"db628d4d-7904-4fdc-91e5-2fec37975821\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " Jul 17 05:52:06.086649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086650 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util\") pod \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " Jul 17 05:52:06.086887 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086669 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkv86\" (UniqueName: \"kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86\") pod \"db628d4d-7904-4fdc-91e5-2fec37975821\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " Jul 17 05:52:06.086887 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086725 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7kf4\" (UniqueName: \"kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4\") pod \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " Jul 17 05:52:06.086887 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086763 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle\") pod \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\" (UID: \"5c2f9e11-811e-41c7-b3dd-b85dc31f0425\") " Jul 17 05:52:06.086887 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.086849 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util\") pod \"db628d4d-7904-4fdc-91e5-2fec37975821\" (UID: \"db628d4d-7904-4fdc-91e5-2fec37975821\") " Jul 17 05:52:06.087345 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.087296 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle" (OuterVolumeSpecName: "bundle") pod "db628d4d-7904-4fdc-91e5-2fec37975821" (UID: "db628d4d-7904-4fdc-91e5-2fec37975821"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:06.087345 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.087304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle" (OuterVolumeSpecName: "bundle") pod "5c2f9e11-811e-41c7-b3dd-b85dc31f0425" (UID: "5c2f9e11-811e-41c7-b3dd-b85dc31f0425"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:06.089435 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.089349 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4" (OuterVolumeSpecName: "kube-api-access-s7kf4") pod "5c2f9e11-811e-41c7-b3dd-b85dc31f0425" (UID: "5c2f9e11-811e-41c7-b3dd-b85dc31f0425"). InnerVolumeSpecName "kube-api-access-s7kf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:52:06.089540 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.089448 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86" (OuterVolumeSpecName: "kube-api-access-pkv86") pod "db628d4d-7904-4fdc-91e5-2fec37975821" (UID: "db628d4d-7904-4fdc-91e5-2fec37975821"). InnerVolumeSpecName "kube-api-access-pkv86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:52:06.093424 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.093400 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util" (OuterVolumeSpecName: "util") pod "db628d4d-7904-4fdc-91e5-2fec37975821" (UID: "db628d4d-7904-4fdc-91e5-2fec37975821"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:06.093541 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.093524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util" (OuterVolumeSpecName: "util") pod "5c2f9e11-811e-41c7-b3dd-b85dc31f0425" (UID: "5c2f9e11-811e-41c7-b3dd-b85dc31f0425"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:06.187738 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187709 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.187738 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187734 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db628d4d-7904-4fdc-91e5-2fec37975821-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.187738 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187742 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.187915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187750 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkv86\" (UniqueName: \"kubernetes.io/projected/db628d4d-7904-4fdc-91e5-2fec37975821-kube-api-access-pkv86\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.187915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187760 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7kf4\" (UniqueName: \"kubernetes.io/projected/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-kube-api-access-s7kf4\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.187915 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.187771 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c2f9e11-811e-41c7-b3dd-b85dc31f0425-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:06.846858 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.846827 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" Jul 17 05:52:06.846858 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.846834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5" event={"ID":"5c2f9e11-811e-41c7-b3dd-b85dc31f0425","Type":"ContainerDied","Data":"8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47"} Jul 17 05:52:06.847323 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.846884 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7350a8d525f7c9906e65d0c98ec2f4beb93ba0abe8f676b8ed7316c8452f47" Jul 17 05:52:06.848738 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.848722 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" Jul 17 05:52:06.848852 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.848753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf" event={"ID":"db628d4d-7904-4fdc-91e5-2fec37975821","Type":"ContainerDied","Data":"65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011"} Jul 17 05:52:06.848852 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.848793 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e1d1588f9cf280b0bc014f91ddfc34bce773f617b3ba1d12a62b5b5a7a0011" Jul 17 05:52:06.851010 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.850985 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerID="71a0408fce7510616ff082f7673830bb2851971efcf92ee71a1b24b0550debb3" exitCode=0 Jul 17 05:52:06.851093 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:06.851013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" event={"ID":"e4a58965-19eb-4337-9b87-f4c81ecc92b4","Type":"ContainerDied","Data":"71a0408fce7510616ff082f7673830bb2851971efcf92ee71a1b24b0550debb3"} Jul 17 05:52:07.992313 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:07.992289 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:08.001891 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.001866 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klqtd\" (UniqueName: \"kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd\") pod \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " Jul 17 05:52:08.001976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.001920 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle\") pod \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " Jul 17 05:52:08.001976 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.001946 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util\") pod \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\" (UID: \"e4a58965-19eb-4337-9b87-f4c81ecc92b4\") " Jul 17 05:52:08.002571 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.002527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle" (OuterVolumeSpecName: "bundle") pod "e4a58965-19eb-4337-9b87-f4c81ecc92b4" (UID: "e4a58965-19eb-4337-9b87-f4c81ecc92b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:08.004139 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.004112 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd" (OuterVolumeSpecName: "kube-api-access-klqtd") pod "e4a58965-19eb-4337-9b87-f4c81ecc92b4" (UID: "e4a58965-19eb-4337-9b87-f4c81ecc92b4"). InnerVolumeSpecName "kube-api-access-klqtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:52:08.007523 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.007497 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util" (OuterVolumeSpecName: "util") pod "e4a58965-19eb-4337-9b87-f4c81ecc92b4" (UID: "e4a58965-19eb-4337-9b87-f4c81ecc92b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 05:52:08.103474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.103426 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klqtd\" (UniqueName: \"kubernetes.io/projected/e4a58965-19eb-4337-9b87-f4c81ecc92b4-kube-api-access-klqtd\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:08.103474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.103469 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:08.103474 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.103480 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a58965-19eb-4337-9b87-f4c81ecc92b4-util\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:52:08.862596 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.862566 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" Jul 17 05:52:08.862596 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.862576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl" event={"ID":"e4a58965-19eb-4337-9b87-f4c81ecc92b4","Type":"ContainerDied","Data":"db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964"} Jul 17 05:52:08.862797 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:08.862619 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db32bec07cbf4240ee066003399163f1c437ba1df521720b9f5c4fc2b43a1964" Jul 17 05:52:35.884731 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.884685 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59df6c8b55-st5v5"] Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.884992 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885003 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885017 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885022 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885029 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885034 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885042 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885047 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885056 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885062 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885068 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885073 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885082 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885087 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885096 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885100 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885107 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885112 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="pull" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885119 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885123 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885130 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885134 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885141 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885145 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="util" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885195 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db628d4d-7904-4fdc-91e5-2fec37975821" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885202 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a58965-19eb-4337-9b87-f4c81ecc92b4" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885207 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2f9e11-811e-41c7-b3dd-b85dc31f0425" containerName="extract" Jul 17 05:52:35.885364 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.885212 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52050454-892f-4b64-b86e-416d3efdf46e" containerName="extract" Jul 17 05:52:35.888036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.888017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:35.898694 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:35.898668 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59df6c8b55-st5v5"] Jul 17 05:52:36.041337 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-console-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041337 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-oauth-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-trusted-ca-bundle\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscbh\" (UniqueName: \"kubernetes.io/projected/45b32a69-423a-41ca-bbb9-773e2094ce81-kube-api-access-mscbh\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-oauth-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.041569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.041510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-service-ca\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142464 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-console-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142464 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-oauth-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142464 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-trusted-ca-bundle\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscbh\" (UniqueName: \"kubernetes.io/projected/45b32a69-423a-41ca-bbb9-773e2094ce81-kube-api-access-mscbh\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-oauth-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.142736 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.142563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-service-ca\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.143192 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.143163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-oauth-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.143348 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.143243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-console-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.143419 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.143375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-trusted-ca-bundle\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.143533 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.143511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/45b32a69-423a-41ca-bbb9-773e2094ce81-service-ca\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.145220 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.145190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-oauth-config\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.145324 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.145293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/45b32a69-423a-41ca-bbb9-773e2094ce81-console-serving-cert\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.154722 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.154703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscbh\" (UniqueName: \"kubernetes.io/projected/45b32a69-423a-41ca-bbb9-773e2094ce81-kube-api-access-mscbh\") pod \"console-59df6c8b55-st5v5\" (UID: \"45b32a69-423a-41ca-bbb9-773e2094ce81\") " pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.197895 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.197874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:36.331951 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:36.331922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59df6c8b55-st5v5"] Jul 17 05:52:36.333807 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:52:36.333781 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b32a69_423a_41ca_bbb9_773e2094ce81.slice/crio-cc2c8bad2f3d20bc590bdff1581b0372192b6a9d67e67d30abd5af2b7128a39e WatchSource:0}: Error finding container cc2c8bad2f3d20bc590bdff1581b0372192b6a9d67e67d30abd5af2b7128a39e: Status 404 returned error can't find the container with id cc2c8bad2f3d20bc590bdff1581b0372192b6a9d67e67d30abd5af2b7128a39e Jul 17 05:52:37.005037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:37.004998 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59df6c8b55-st5v5" event={"ID":"45b32a69-423a-41ca-bbb9-773e2094ce81","Type":"ContainerStarted","Data":"616936a27866a67347bdd976acdbff79dc405704b301332c8e539de2579937d9"} Jul 17 05:52:37.005449 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:37.005052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59df6c8b55-st5v5" event={"ID":"45b32a69-423a-41ca-bbb9-773e2094ce81","Type":"ContainerStarted","Data":"cc2c8bad2f3d20bc590bdff1581b0372192b6a9d67e67d30abd5af2b7128a39e"} Jul 17 05:52:37.024577 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:37.024529 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59df6c8b55-st5v5" podStartSLOduration=2.024514507 podStartE2EDuration="2.024514507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:52:37.022920176 +0000 UTC m=+684.212300462" watchObservedRunningTime="2026-07-17 05:52:37.024514507 +0000 UTC m=+684.213894789" Jul 17 05:52:46.198004 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:46.197967 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:46.198504 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:46.198032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:46.202707 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:46.202686 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:47.062340 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:47.062311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59df6c8b55-st5v5" Jul 17 05:52:47.115836 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:52:47.115802 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:53:12.143244 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.143158 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64774445fb-sp257" podUID="5b6a41d2-865b-4eee-be71-a475ac38ab60" containerName="console" containerID="cri-o://a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc" gracePeriod=15 Jul 17 05:53:12.384055 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.384032 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:53:12.433702 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433702 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433651 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw2g6\" (UniqueName: \"kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433702 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433686 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433719 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433752 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433780 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.433981 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.433843 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert\") pod \"5b6a41d2-865b-4eee-be71-a475ac38ab60\" (UID: \"5b6a41d2-865b-4eee-be71-a475ac38ab60\") " Jul 17 05:53:12.434231 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.434157 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:53:12.434231 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.434191 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:53:12.434231 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.434199 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config" (OuterVolumeSpecName: "console-config") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:53:12.434630 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.434509 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:53:12.436065 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.436038 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:53:12.436065 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.436052 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 05:53:12.436197 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.436095 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6" (OuterVolumeSpecName: "kube-api-access-lw2g6") pod "5b6a41d2-865b-4eee-be71-a475ac38ab60" (UID: "5b6a41d2-865b-4eee-be71-a475ac38ab60"). InnerVolumeSpecName "kube-api-access-lw2g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:53:12.534880 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534840 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.534880 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534874 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.534880 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534884 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw2g6\" (UniqueName: \"kubernetes.io/projected/5b6a41d2-865b-4eee-be71-a475ac38ab60-kube-api-access-lw2g6\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.535115 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534893 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.535115 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534902 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.535115 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534910 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:12.535115 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:12.534917 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b6a41d2-865b-4eee-be71-a475ac38ab60-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:13.204959 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.204924 2574 generic.go:356] "Generic (PLEG): container finished" podID="5b6a41d2-865b-4eee-be71-a475ac38ab60" containerID="a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc" exitCode=0 Jul 17 05:53:13.205367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.204974 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64774445fb-sp257" Jul 17 05:53:13.205367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.205014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64774445fb-sp257" event={"ID":"5b6a41d2-865b-4eee-be71-a475ac38ab60","Type":"ContainerDied","Data":"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc"} Jul 17 05:53:13.205367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.205095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64774445fb-sp257" event={"ID":"5b6a41d2-865b-4eee-be71-a475ac38ab60","Type":"ContainerDied","Data":"81e8ca89ab1fbb1af7fb29fe5a60ee02d3439ffbe40c6eced092535124e18b1e"} Jul 17 05:53:13.205367 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.205120 2574 scope.go:117] "RemoveContainer" containerID="a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc" Jul 17 05:53:13.214555 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.214539 2574 scope.go:117] "RemoveContainer" containerID="a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc" Jul 17 05:53:13.214809 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:53:13.214790 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc\": container with ID starting with a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc not found: ID does not exist" containerID="a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc" Jul 17 05:53:13.214853 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.214819 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc"} err="failed to get container status \"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc\": rpc error: code = NotFound desc = could not find container \"a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc\": container with ID starting with a05f98c72693c75d1dcef9b542013be5b450e76507255ecbfdc3f775f3a225cc not found: ID does not exist" Jul 17 05:53:13.224966 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.224941 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:53:13.228796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.228775 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64774445fb-sp257"] Jul 17 05:53:13.282497 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:13.282471 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6a41d2-865b-4eee-be71-a475ac38ab60" path="/var/lib/kubelet/pods/5b6a41d2-865b-4eee-be71-a475ac38ab60/volumes" Jul 17 05:53:41.959994 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.959959 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:41.960419 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.960328 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6a41d2-865b-4eee-be71-a475ac38ab60" containerName="console" Jul 17 05:53:41.960419 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.960347 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6a41d2-865b-4eee-be71-a475ac38ab60" containerName="console" Jul 17 05:53:41.960419 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.960414 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6a41d2-865b-4eee-be71-a475ac38ab60" containerName="console" Jul 17 05:53:41.963331 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.963315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:41.965507 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.965487 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 05:53:41.965616 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.965560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cksw5\"" Jul 17 05:53:41.966519 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.966496 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 05:53:41.966616 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.966502 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 05:53:41.970050 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:41.970032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:42.067320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.067287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:42.084298 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.084241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtblj\" (UniqueName: \"kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.084445 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.084339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.185680 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.185648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtblj\" (UniqueName: \"kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.185855 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.185701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.186350 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.186333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.194757 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.194737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtblj\" (UniqueName: \"kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj\") pod \"limitador-limitador-64c94fd64f-rwprt\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.274610 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.274526 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:42.409680 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.409656 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:42.411640 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:53:42.411604 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e489353_1f8c_43b4_9175_e309e5344859.slice/crio-7b4dec84ef3bb080c7d0d9e77687f908f9badc646f1f89dc600675c1879f5e62 WatchSource:0}: Error finding container 7b4dec84ef3bb080c7d0d9e77687f908f9badc646f1f89dc600675c1879f5e62: Status 404 returned error can't find the container with id 7b4dec84ef3bb080c7d0d9e77687f908f9badc646f1f89dc600675c1879f5e62 Jul 17 05:53:42.413701 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.413686 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 05:53:42.929680 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.929649 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:53:42.933918 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.933901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:53:42.936360 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.936332 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5b7zn\"" Jul 17 05:53:42.938871 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:42.938851 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:53:43.093796 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.093765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmbw\" (UniqueName: \"kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw\") pod \"authorino-75d556bd77-mvqvz\" (UID: \"c6cbb8b9-2a65-4522-bcee-e64df79b3879\") " pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:53:43.194649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.194563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmbw\" (UniqueName: \"kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw\") pod \"authorino-75d556bd77-mvqvz\" (UID: \"c6cbb8b9-2a65-4522-bcee-e64df79b3879\") " pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:53:43.202356 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.202323 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmbw\" (UniqueName: \"kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw\") pod \"authorino-75d556bd77-mvqvz\" (UID: \"c6cbb8b9-2a65-4522-bcee-e64df79b3879\") " pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:53:43.244491 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.244458 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:53:43.358431 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.358392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" event={"ID":"5e489353-1f8c-43b4-9175-e309e5344859","Type":"ContainerStarted","Data":"7b4dec84ef3bb080c7d0d9e77687f908f9badc646f1f89dc600675c1879f5e62"} Jul 17 05:53:43.436857 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:43.436841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:53:43.444721 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:53:43.444644 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6cbb8b9_2a65_4522_bcee_e64df79b3879.slice/crio-2cc64579bb1493c7bbb1003e639a71e6edd947843f5ef7753628de59fbe1e6a5 WatchSource:0}: Error finding container 2cc64579bb1493c7bbb1003e639a71e6edd947843f5ef7753628de59fbe1e6a5: Status 404 returned error can't find the container with id 2cc64579bb1493c7bbb1003e639a71e6edd947843f5ef7753628de59fbe1e6a5 Jul 17 05:53:44.380755 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:44.380712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mvqvz" event={"ID":"c6cbb8b9-2a65-4522-bcee-e64df79b3879","Type":"ContainerStarted","Data":"2cc64579bb1493c7bbb1003e639a71e6edd947843f5ef7753628de59fbe1e6a5"} Jul 17 05:53:47.398504 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:47.398463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" event={"ID":"5e489353-1f8c-43b4-9175-e309e5344859","Type":"ContainerStarted","Data":"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff"} Jul 17 05:53:47.398939 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:47.398633 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:47.400068 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:47.400044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mvqvz" event={"ID":"c6cbb8b9-2a65-4522-bcee-e64df79b3879","Type":"ContainerStarted","Data":"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61"} Jul 17 05:53:47.414009 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:47.413966 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" podStartSLOduration=1.6147765459999999 podStartE2EDuration="6.413949554s" totalImagesPullingTime="4.799173008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:53:47.413432818 +0000 UTC m=+754.602813125" watchObservedRunningTime="2026-07-17 05:53:47.413949554 +0000 UTC m=+754.603329838" Jul 17 05:53:47.427571 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:47.427531 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-mvqvz" podStartSLOduration=1.6589664119999998 podStartE2EDuration="5.427519581s" totalImagesPullingTime="3.768553169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:53:47.42570791 +0000 UTC m=+754.615088187" watchObservedRunningTime="2026-07-17 05:53:47.427519581 +0000 UTC m=+754.616899864" Jul 17 05:53:56.642698 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:56.642662 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:56.643178 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:56.642861 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" podUID="5e489353-1f8c-43b4-9175-e309e5344859" containerName="limitador" containerID="cri-o://a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff" gracePeriod=30 Jul 17 05:53:56.643649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:56.643632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:57.181115 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.181093 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:57.217972 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.217884 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtblj\" (UniqueName: \"kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj\") pod \"5e489353-1f8c-43b4-9175-e309e5344859\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " Jul 17 05:53:57.217972 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.217963 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file\") pod \"5e489353-1f8c-43b4-9175-e309e5344859\" (UID: \"5e489353-1f8c-43b4-9175-e309e5344859\") " Jul 17 05:53:57.218306 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.218282 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file" (OuterVolumeSpecName: "config-file") pod "5e489353-1f8c-43b4-9175-e309e5344859" (UID: "5e489353-1f8c-43b4-9175-e309e5344859"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 05:53:57.220091 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.220057 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj" (OuterVolumeSpecName: "kube-api-access-mtblj") pod "5e489353-1f8c-43b4-9175-e309e5344859" (UID: "5e489353-1f8c-43b4-9175-e309e5344859"). InnerVolumeSpecName "kube-api-access-mtblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:53:57.318587 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.318562 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5e489353-1f8c-43b4-9175-e309e5344859-config-file\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:57.318694 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.318604 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtblj\" (UniqueName: \"kubernetes.io/projected/5e489353-1f8c-43b4-9175-e309e5344859-kube-api-access-mtblj\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:53:57.450379 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.450346 2574 generic.go:356] "Generic (PLEG): container finished" podID="5e489353-1f8c-43b4-9175-e309e5344859" containerID="a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff" exitCode=0 Jul 17 05:53:57.450539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.450391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" event={"ID":"5e489353-1f8c-43b4-9175-e309e5344859","Type":"ContainerDied","Data":"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff"} Jul 17 05:53:57.450539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.450413 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" Jul 17 05:53:57.450539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.450454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rwprt" event={"ID":"5e489353-1f8c-43b4-9175-e309e5344859","Type":"ContainerDied","Data":"7b4dec84ef3bb080c7d0d9e77687f908f9badc646f1f89dc600675c1879f5e62"} Jul 17 05:53:57.450539 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.450476 2574 scope.go:117] "RemoveContainer" containerID="a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff" Jul 17 05:53:57.459497 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.459478 2574 scope.go:117] "RemoveContainer" containerID="a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff" Jul 17 05:53:57.459742 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:53:57.459723 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff\": container with ID starting with a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff not found: ID does not exist" containerID="a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff" Jul 17 05:53:57.459784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.459750 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff"} err="failed to get container status \"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff\": rpc error: code = NotFound desc = could not find container \"a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff\": container with ID starting with a2a7b4897679a7a57e2995139af49dbfc03d2c5b65cd261b71742de5862311ff not found: ID does not exist" Jul 17 05:53:57.468730 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.468678 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:57.471944 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:57.471924 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rwprt"] Jul 17 05:53:59.282197 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.282166 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e489353-1f8c-43b4-9175-e309e5344859" path="/var/lib/kubelet/pods/5e489353-1f8c-43b4-9175-e309e5344859/volumes" Jul 17 05:53:59.804580 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.804538 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-864zj"] Jul 17 05:53:59.805099 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.805074 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e489353-1f8c-43b4-9175-e309e5344859" containerName="limitador" Jul 17 05:53:59.805266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.805194 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e489353-1f8c-43b4-9175-e309e5344859" containerName="limitador" Jul 17 05:53:59.805353 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.805328 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e489353-1f8c-43b4-9175-e309e5344859" containerName="limitador" Jul 17 05:53:59.810046 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.810024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.812593 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.812571 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 05:53:59.812719 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.812670 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dvfln\"" Jul 17 05:53:59.812785 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.812756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 17 05:53:59.812827 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.812813 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 05:53:59.818629 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.818605 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-864zj"] Jul 17 05:53:59.838090 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.838065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86a8f91b-b0be-42e8-b473-9dd1f30750d7-data\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.838206 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.838096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptx24\" (UniqueName: \"kubernetes.io/projected/86a8f91b-b0be-42e8-b473-9dd1f30750d7-kube-api-access-ptx24\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.938891 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.938868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86a8f91b-b0be-42e8-b473-9dd1f30750d7-data\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.939037 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.938898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptx24\" (UniqueName: \"kubernetes.io/projected/86a8f91b-b0be-42e8-b473-9dd1f30750d7-kube-api-access-ptx24\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.939231 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.939213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/86a8f91b-b0be-42e8-b473-9dd1f30750d7-data\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:53:59.950193 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:53:59.950176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptx24\" (UniqueName: \"kubernetes.io/projected/86a8f91b-b0be-42e8-b473-9dd1f30750d7-kube-api-access-ptx24\") pod \"postgres-69fb4594cf-864zj\" (UID: \"86a8f91b-b0be-42e8-b473-9dd1f30750d7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:54:00.121928 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:00.121892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:54:00.267036 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:00.267004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-864zj"] Jul 17 05:54:00.270209 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:54:00.270178 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a8f91b_b0be_42e8_b473_9dd1f30750d7.slice/crio-086e17d59310e14ca4ae544b9b9fcf2a91cbf0ee0fc0eb812f8df98df1a4490a WatchSource:0}: Error finding container 086e17d59310e14ca4ae544b9b9fcf2a91cbf0ee0fc0eb812f8df98df1a4490a: Status 404 returned error can't find the container with id 086e17d59310e14ca4ae544b9b9fcf2a91cbf0ee0fc0eb812f8df98df1a4490a Jul 17 05:54:00.468050 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:00.467961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" event={"ID":"86a8f91b-b0be-42e8-b473-9dd1f30750d7","Type":"ContainerStarted","Data":"086e17d59310e14ca4ae544b9b9fcf2a91cbf0ee0fc0eb812f8df98df1a4490a"} Jul 17 05:54:05.500776 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:05.500686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" event={"ID":"86a8f91b-b0be-42e8-b473-9dd1f30750d7","Type":"ContainerStarted","Data":"239f97c0a95336529e6e60111e369b2867ceeed33346bcff617e74992d8797b8"} Jul 17 05:54:05.501140 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:05.500782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:54:05.516378 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:05.516329 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" podStartSLOduration=1.673724804 podStartE2EDuration="6.516314253s" totalImagesPullingTime="4.842589449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:54:05.515581594 +0000 UTC m=+772.704961877" watchObservedRunningTime="2026-07-17 05:54:05.516314253 +0000 UTC m=+772.705694565" Jul 17 05:54:11.536372 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:11.536336 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-864zj" Jul 17 05:54:14.062721 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.062683 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:54:14.063199 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.062851 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-mvqvz" podUID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" containerName="authorino" containerID="cri-o://bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61" gracePeriod=30 Jul 17 05:54:14.345065 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.345041 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:54:14.462782 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.462753 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmmbw\" (UniqueName: \"kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw\") pod \"c6cbb8b9-2a65-4522-bcee-e64df79b3879\" (UID: \"c6cbb8b9-2a65-4522-bcee-e64df79b3879\") " Jul 17 05:54:14.464978 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.464942 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw" (OuterVolumeSpecName: "kube-api-access-vmmbw") pod "c6cbb8b9-2a65-4522-bcee-e64df79b3879" (UID: "c6cbb8b9-2a65-4522-bcee-e64df79b3879"). InnerVolumeSpecName "kube-api-access-vmmbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 05:54:14.548063 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.548030 2574 generic.go:356] "Generic (PLEG): container finished" podID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" containerID="bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61" exitCode=0 Jul 17 05:54:14.548233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.548075 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-mvqvz" Jul 17 05:54:14.548233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.548123 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mvqvz" event={"ID":"c6cbb8b9-2a65-4522-bcee-e64df79b3879","Type":"ContainerDied","Data":"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61"} Jul 17 05:54:14.548233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.548202 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-mvqvz" event={"ID":"c6cbb8b9-2a65-4522-bcee-e64df79b3879","Type":"ContainerDied","Data":"2cc64579bb1493c7bbb1003e639a71e6edd947843f5ef7753628de59fbe1e6a5"} Jul 17 05:54:14.548233 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.548228 2574 scope.go:117] "RemoveContainer" containerID="bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61" Jul 17 05:54:14.557516 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.557500 2574 scope.go:117] "RemoveContainer" containerID="bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61" Jul 17 05:54:14.557772 ip-10-0-136-88 kubenswrapper[2574]: E0717 05:54:14.557751 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61\": container with ID starting with bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61 not found: ID does not exist" containerID="bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61" Jul 17 05:54:14.557843 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.557786 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61"} err="failed to get container status \"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61\": rpc error: code = NotFound desc = could not find container \"bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61\": container with ID starting with bcd594f237d69d30d59727cccda4cf0e4453aba3783afe504755bce45c269e61 not found: ID does not exist" Jul 17 05:54:14.564075 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.564052 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmbw\" (UniqueName: \"kubernetes.io/projected/c6cbb8b9-2a65-4522-bcee-e64df79b3879-kube-api-access-vmmbw\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 05:54:14.568374 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.568324 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:54:14.570196 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:14.570178 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-mvqvz"] Jul 17 05:54:15.282219 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:15.282191 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" path="/var/lib/kubelet/pods/c6cbb8b9-2a65-4522-bcee-e64df79b3879/volumes" Jul 17 05:54:56.216017 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.215979 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bkl69"] Jul 17 05:54:56.217438 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.217412 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" containerName="authorino" Jul 17 05:54:56.217597 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.217585 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" containerName="authorino" Jul 17 05:54:56.217845 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.217831 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6cbb8b9-2a65-4522-bcee-e64df79b3879" containerName="authorino" Jul 17 05:54:56.221262 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.221225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.223726 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.223704 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 05:54:56.223818 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.223789 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5klkl\"" Jul 17 05:54:56.230290 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.230152 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bkl69"] Jul 17 05:54:56.232631 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.232610 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tprvq"] Jul 17 05:54:56.234883 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.234866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.246713 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.246689 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tprvq"] Jul 17 05:54:56.320756 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.320728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pq8\" (UniqueName: \"kubernetes.io/projected/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-kube-api-access-56pq8\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.320911 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.320774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67mk\" (UniqueName: \"kubernetes.io/projected/b39145f8-c13e-4957-83d2-3acf8555813a-kube-api-access-h67mk\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.320911 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.320811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.320911 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.320889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39145f8-c13e-4957-83d2-3acf8555813a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.421744 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.421714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pq8\" (UniqueName: \"kubernetes.io/projected/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-kube-api-access-56pq8\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.421902 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.421753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67mk\" (UniqueName: \"kubernetes.io/projected/b39145f8-c13e-4957-83d2-3acf8555813a-kube-api-access-h67mk\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.421902 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.421775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.422020 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.421905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39145f8-c13e-4957-83d2-3acf8555813a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.422525 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.422508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.422578 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.422561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39145f8-c13e-4957-83d2-3acf8555813a-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.430667 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.430644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67mk\" (UniqueName: \"kubernetes.io/projected/b39145f8-c13e-4957-83d2-3acf8555813a-kube-api-access-h67mk\") pod \"payload-processing-5bf47cd988-tprvq\" (UID: \"b39145f8-c13e-4957-83d2-3acf8555813a\") " pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.430765 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.430732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pq8\" (UniqueName: \"kubernetes.io/projected/a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af-kube-api-access-56pq8\") pod \"payload-pre-processing-7677b774bb-bkl69\" (UID: \"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.532848 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.532774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:56.544834 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.544811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:56.675131 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.675105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-bkl69"] Jul 17 05:54:56.677165 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:54:56.677134 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44f66c3_f8c5_4fe2_b5ce_7b0af4a775af.slice/crio-360522707f368b5a4b2e9348a5c947b16c2deb7110e3c752692f4e922d0a5258 WatchSource:0}: Error finding container 360522707f368b5a4b2e9348a5c947b16c2deb7110e3c752692f4e922d0a5258: Status 404 returned error can't find the container with id 360522707f368b5a4b2e9348a5c947b16c2deb7110e3c752692f4e922d0a5258 Jul 17 05:54:56.701206 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.701177 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tprvq"] Jul 17 05:54:56.702209 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:54:56.702185 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39145f8_c13e_4957_83d2_3acf8555813a.slice/crio-db7d53f68a8c714abe15ff57288f7dbba6e4ca890418c557a66bbe3baf7a2560 WatchSource:0}: Error finding container db7d53f68a8c714abe15ff57288f7dbba6e4ca890418c557a66bbe3baf7a2560: Status 404 returned error can't find the container with id db7d53f68a8c714abe15ff57288f7dbba6e4ca890418c557a66bbe3baf7a2560 Jul 17 05:54:56.764155 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.764127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" event={"ID":"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af","Type":"ContainerStarted","Data":"360522707f368b5a4b2e9348a5c947b16c2deb7110e3c752692f4e922d0a5258"} Jul 17 05:54:56.765336 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:56.765296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" event={"ID":"b39145f8-c13e-4957-83d2-3acf8555813a","Type":"ContainerStarted","Data":"db7d53f68a8c714abe15ff57288f7dbba6e4ca890418c557a66bbe3baf7a2560"} Jul 17 05:54:58.779180 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.779086 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" event={"ID":"a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af","Type":"ContainerStarted","Data":"81ec270a119351f51cc5e5b5edf0d48a11980aefcb6c2c5e3135aab7adbd6895"} Jul 17 05:54:58.779649 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.779240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:54:58.780846 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.780821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" event={"ID":"b39145f8-c13e-4957-83d2-3acf8555813a","Type":"ContainerStarted","Data":"3bed06e619b6d6779a47cb5f9e8b9f15a5c320bad726156640c30420dadf0e2f"} Jul 17 05:54:58.780959 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.780884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:54:58.795601 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.795547 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" podStartSLOduration=1.065319018 podStartE2EDuration="2.795528389s" totalImagesPullingTime="1.730209371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:54:58.793089282 +0000 UTC m=+825.982469566" watchObservedRunningTime="2026-07-17 05:54:58.795528389 +0000 UTC m=+825.984908674" Jul 17 05:54:58.807266 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:54:58.807204 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" podStartSLOduration=1.091760513 podStartE2EDuration="2.807191018s" totalImagesPullingTime="1.715430505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:54:58.805816287 +0000 UTC m=+825.995196573" watchObservedRunningTime="2026-07-17 05:54:58.807191018 +0000 UTC m=+825.996571300" Jul 17 05:55:09.788269 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:09.788229 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tprvq" Jul 17 05:55:09.788642 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:09.788547 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-bkl69" Jul 17 05:55:42.881274 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.881213 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh"] Jul 17 05:55:42.884642 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.884622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.887075 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.887047 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 05:55:42.888006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.887987 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 17 05:55:42.888006 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.888011 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8bkch\"" Jul 17 05:55:42.888190 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.888043 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 05:55:42.893579 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.893555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh"] Jul 17 05:55:42.912030 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.912153 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zh6\" (UniqueName: \"kubernetes.io/projected/ebd9065b-f376-47c9-a27d-0efed5c1b427-kube-api-access-s6zh6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.912153 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd9065b-f376-47c9-a27d-0efed5c1b427-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.912153 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.912153 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:42.912352 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:42.912204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013003 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.012973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zh6\" (UniqueName: \"kubernetes.io/projected/ebd9065b-f376-47c9-a27d-0efed5c1b427-kube-api-access-s6zh6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd9065b-f376-47c9-a27d-0efed5c1b427-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013189 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013506 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013569 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.013626 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.013586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.015530 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.015510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd9065b-f376-47c9-a27d-0efed5c1b427-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.015692 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.015676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd9065b-f376-47c9-a27d-0efed5c1b427-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.020906 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.020888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zh6\" (UniqueName: \"kubernetes.io/projected/ebd9065b-f376-47c9-a27d-0efed5c1b427-kube-api-access-s6zh6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vvfnh\" (UID: \"ebd9065b-f376-47c9-a27d-0efed5c1b427\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.196110 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.196008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:43.335866 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:43.335838 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh"] Jul 17 05:55:43.337245 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:55:43.337215 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd9065b_f376_47c9_a27d_0efed5c1b427.slice/crio-7fbb988346a8d08d5affd083e45232ba027bb74d35687d6ca74b8890da03f47a WatchSource:0}: Error finding container 7fbb988346a8d08d5affd083e45232ba027bb74d35687d6ca74b8890da03f47a: Status 404 returned error can't find the container with id 7fbb988346a8d08d5affd083e45232ba027bb74d35687d6ca74b8890da03f47a Jul 17 05:55:44.017140 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:44.017103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" event={"ID":"ebd9065b-f376-47c9-a27d-0efed5c1b427","Type":"ContainerStarted","Data":"7fbb988346a8d08d5affd083e45232ba027bb74d35687d6ca74b8890da03f47a"} Jul 17 05:55:45.024879 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:45.024843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" event={"ID":"ebd9065b-f376-47c9-a27d-0efed5c1b427","Type":"ContainerStarted","Data":"2def25a883a43dd967948851cf6ef05d89f4f091640336828d6711681fef2413"} Jul 17 05:55:45.041713 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:45.041664 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" podStartSLOduration=2.021109839 podStartE2EDuration="3.041650439s" totalImagesPullingTime="1.0205406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:55:45.040215082 +0000 UTC m=+872.229595367" watchObservedRunningTime="2026-07-17 05:55:45.041650439 +0000 UTC m=+872.231030723" Jul 17 05:55:53.197102 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:53.197071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:53.197527 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:53.197128 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:53.209589 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:53.209565 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:55:54.080332 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:55:54.080304 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vvfnh" Jul 17 05:56:13.487508 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:13.487417 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:56:13.502797 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:13.488695 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 05:56:49.770784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.770748 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l"] Jul 17 05:56:49.774550 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.774528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.776912 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.776893 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 05:56:49.783766 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.783722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l"] Jul 17 05:56:49.875925 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.875899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdws\" (UniqueName: \"kubernetes.io/projected/67329077-a9e8-433b-8f93-4b03cde5e357-kube-api-access-fqdws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.876084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.875933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.876084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.875961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.876084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.875982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.876084 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.876053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.876235 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.876085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67329077-a9e8-433b-8f93-4b03cde5e357-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977453 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdws\" (UniqueName: \"kubernetes.io/projected/67329077-a9e8-433b-8f93-4b03cde5e357-kube-api-access-fqdws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977646 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67329077-a9e8-433b-8f93-4b03cde5e357-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977899 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.977899 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.978007 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.977928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.980014 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.979992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67329077-a9e8-433b-8f93-4b03cde5e357-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.980305 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.980287 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67329077-a9e8-433b-8f93-4b03cde5e357-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:49.984592 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:49.984574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdws\" (UniqueName: \"kubernetes.io/projected/67329077-a9e8-433b-8f93-4b03cde5e357-kube-api-access-fqdws\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l\" (UID: \"67329077-a9e8-433b-8f93-4b03cde5e357\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:50.084591 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:50.084559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:56:50.216304 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:50.216276 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l"] Jul 17 05:56:50.219217 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:56:50.219191 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67329077_a9e8_433b_8f93_4b03cde5e357.slice/crio-48f1692bc74792c3a3c0cad762cbdd1081647618f8349eb401bd8ca26d06b713 WatchSource:0}: Error finding container 48f1692bc74792c3a3c0cad762cbdd1081647618f8349eb401bd8ca26d06b713: Status 404 returned error can't find the container with id 48f1692bc74792c3a3c0cad762cbdd1081647618f8349eb401bd8ca26d06b713 Jul 17 05:56:50.360970 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:50.360900 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" event={"ID":"67329077-a9e8-433b-8f93-4b03cde5e357","Type":"ContainerStarted","Data":"48f1692bc74792c3a3c0cad762cbdd1081647618f8349eb401bd8ca26d06b713"} Jul 17 05:56:51.367790 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:51.367752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" event={"ID":"67329077-a9e8-433b-8f93-4b03cde5e357","Type":"ContainerStarted","Data":"ed5c34dfa2164b790c0d31395527c36e92ecbe307addc03ee6287234fd3bbce7"} Jul 17 05:56:51.385471 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:56:51.385421 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" podStartSLOduration=2.18052325 podStartE2EDuration="2.385404235s" totalImagesPullingTime="204.880985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:56:51.382896102 +0000 UTC m=+938.572276384" watchObservedRunningTime="2026-07-17 05:56:51.385404235 +0000 UTC m=+938.574784519" Jul 17 05:57:00.085201 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:57:00.085139 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:57:00.085704 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:57:00.085230 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:57:00.097951 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:57:00.097926 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:57:00.424319 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:57:00.424225 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l" Jul 17 05:58:09.289583 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.289555 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5774d84fdb-qdzd9"] Jul 17 05:58:09.292809 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.292792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.296090 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.296059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 05:58:09.296236 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.296059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5b7zn\"" Jul 17 05:58:09.296236 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.296063 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 05:58:09.296236 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.296068 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 05:58:09.302583 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.302564 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5774d84fdb-qdzd9"] Jul 17 05:58:09.445122 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.445090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6mx\" (UniqueName: \"kubernetes.io/projected/456064d9-05e9-433a-8055-6e917444af36-kube-api-access-9x6mx\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.445320 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.445128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/456064d9-05e9-433a-8055-6e917444af36-tls-cert\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.545784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.545722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6mx\" (UniqueName: \"kubernetes.io/projected/456064d9-05e9-433a-8055-6e917444af36-kube-api-access-9x6mx\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.545784 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.545756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/456064d9-05e9-433a-8055-6e917444af36-tls-cert\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.548375 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.548349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/456064d9-05e9-433a-8055-6e917444af36-tls-cert\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.552908 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.552891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6mx\" (UniqueName: \"kubernetes.io/projected/456064d9-05e9-433a-8055-6e917444af36-kube-api-access-9x6mx\") pod \"authorino-5774d84fdb-qdzd9\" (UID: \"456064d9-05e9-433a-8055-6e917444af36\") " pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.602884 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.602862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5774d84fdb-qdzd9" Jul 17 05:58:09.758380 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.758346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5774d84fdb-qdzd9"] Jul 17 05:58:09.759749 ip-10-0-136-88 kubenswrapper[2574]: W0717 05:58:09.759724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod456064d9_05e9_433a_8055_6e917444af36.slice/crio-cd85b0d25dd8bcaf7f1fc8778edd2a2eb506c7e0346912e927ec0a597fefcedc WatchSource:0}: Error finding container cd85b0d25dd8bcaf7f1fc8778edd2a2eb506c7e0346912e927ec0a597fefcedc: Status 404 returned error can't find the container with id cd85b0d25dd8bcaf7f1fc8778edd2a2eb506c7e0346912e927ec0a597fefcedc Jul 17 05:58:09.779667 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:09.779643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5774d84fdb-qdzd9" event={"ID":"456064d9-05e9-433a-8055-6e917444af36","Type":"ContainerStarted","Data":"cd85b0d25dd8bcaf7f1fc8778edd2a2eb506c7e0346912e927ec0a597fefcedc"} Jul 17 05:58:10.787703 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:10.787624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5774d84fdb-qdzd9" event={"ID":"456064d9-05e9-433a-8055-6e917444af36","Type":"ContainerStarted","Data":"47904cfd37f863d54ea2c9ae32e03ddcc5500d823d8e3fe1ff532a52f967da76"} Jul 17 05:58:10.803613 ip-10-0-136-88 kubenswrapper[2574]: I0717 05:58:10.803562 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5774d84fdb-qdzd9" podStartSLOduration=1.204548064 podStartE2EDuration="1.80354516s" totalImagesPullingTime="598.997096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 05:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 05:58:10.802839789 +0000 UTC m=+1017.992220072" watchObservedRunningTime="2026-07-17 05:58:10.80354516 +0000 UTC m=+1017.992925443" Jul 17 06:01:13.527474 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:01:13.527354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:01:13.531333 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:01:13.531023 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:06:13.567234 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:06:13.567128 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:06:13.573888 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:06:13.573867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:11:13.603849 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:11:13.603818 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:11:13.613319 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:11:13.613297 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:13:05.210429 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.210392 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:05.213499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.213482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:05.215721 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.215703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:13:05.220726 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.220698 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:05.382800 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.382771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-d2xls\" (UID: \"977b3c0a-ac76-43c9-867a-b8215c91fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:05.483702 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.483624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-d2xls\" (UID: \"977b3c0a-ac76-43c9-867a-b8215c91fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:05.484202 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.484184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-d2xls\" (UID: \"977b3c0a-ac76-43c9-867a-b8215c91fd36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:05.524682 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.524663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:05.657740 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.657713 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:05.659129 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:13:05.659099 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977b3c0a_ac76_43c9_867a_b8215c91fd36.slice/crio-52f411b844c064984219fe86da6e4b2e55bb1eb2d624a905a5b6d600c048dc57 WatchSource:0}: Error finding container 52f411b844c064984219fe86da6e4b2e55bb1eb2d624a905a5b6d600c048dc57: Status 404 returned error can't find the container with id 52f411b844c064984219fe86da6e4b2e55bb1eb2d624a905a5b6d600c048dc57 Jul 17 06:13:05.661095 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:05.661077 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:13:06.518192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:06.518160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerStarted","Data":"52f411b844c064984219fe86da6e4b2e55bb1eb2d624a905a5b6d600c048dc57"} Jul 17 06:13:08.407467 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:13:08.407431 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977b3c0a_ac76_43c9_867a_b8215c91fd36.slice/crio-00a8e03b23928bfc6f18e515d6d25857f1cd5b0f20ce4b6d7e3dd01bae653f0e.scope\": RecentStats: unable to find data in memory cache]" Jul 17 06:13:08.530635 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:08.530602 2574 generic.go:356] "Generic (PLEG): container finished" podID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerID="00a8e03b23928bfc6f18e515d6d25857f1cd5b0f20ce4b6d7e3dd01bae653f0e" exitCode=7 Jul 17 06:13:08.530787 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:08.530664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerDied","Data":"00a8e03b23928bfc6f18e515d6d25857f1cd5b0f20ce4b6d7e3dd01bae653f0e"} Jul 17 06:13:08.530863 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:08.530852 2574 scope.go:117] "RemoveContainer" containerID="00a8e03b23928bfc6f18e515d6d25857f1cd5b0f20ce4b6d7e3dd01bae653f0e" Jul 17 06:13:09.537022 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:09.536981 2574 generic.go:356] "Generic (PLEG): container finished" podID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerID="445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08" exitCode=7 Jul 17 06:13:09.537459 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:09.537075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerDied","Data":"445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08"} Jul 17 06:13:09.537459 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:09.537145 2574 scope.go:117] "RemoveContainer" containerID="00a8e03b23928bfc6f18e515d6d25857f1cd5b0f20ce4b6d7e3dd01bae653f0e" Jul 17 06:13:09.537459 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:09.537345 2574 scope.go:117] "RemoveContainer" containerID="445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08" Jul 17 06:13:09.537630 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:13:09.537600 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-d2xls_odh-ai-gateway-infra(977b3c0a-ac76-43c9-867a-b8215c91fd36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" Jul 17 06:13:10.544988 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:10.544963 2574 scope.go:117] "RemoveContainer" containerID="445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08" Jul 17 06:13:10.545393 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:13:10.545137 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-d2xls_odh-ai-gateway-infra(977b3c0a-ac76-43c9-867a-b8215c91fd36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" Jul 17 06:13:24.276948 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:24.276915 2574 scope.go:117] "RemoveContainer" containerID="445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08" Jul 17 06:13:24.617243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:24.617207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerStarted","Data":"1c60e035c6e02cfc8de1b1656bcf075bf72f460c1222f8f3c081265ab4a7e028"} Jul 17 06:13:24.632885 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:24.632837 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" podStartSLOduration=17.467028854 podStartE2EDuration="19.632821766s" totalImagesPullingTime="2.165792912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:13:24.630979896 +0000 UTC m=+1931.820360180" watchObservedRunningTime="2026-07-17 06:13:24.632821766 +0000 UTC m=+1931.822202049" Jul 17 06:13:25.623936 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:25.623906 2574 generic.go:356] "Generic (PLEG): container finished" podID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerID="1c60e035c6e02cfc8de1b1656bcf075bf72f460c1222f8f3c081265ab4a7e028" exitCode=7 Jul 17 06:13:25.624407 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:25.623994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerDied","Data":"1c60e035c6e02cfc8de1b1656bcf075bf72f460c1222f8f3c081265ab4a7e028"} Jul 17 06:13:25.624407 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:25.624059 2574 scope.go:117] "RemoveContainer" containerID="445be1b520942ae0a7e9bf255cc3e724613aff3fcc8c29bc4fff4a9f14ecca08" Jul 17 06:13:25.624407 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:25.624360 2574 scope.go:117] "RemoveContainer" containerID="1c60e035c6e02cfc8de1b1656bcf075bf72f460c1222f8f3c081265ab4a7e028" Jul 17 06:13:25.624628 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:13:25.624609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-d2xls_odh-ai-gateway-infra(977b3c0a-ac76-43c9-867a-b8215c91fd36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" Jul 17 06:13:25.647624 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:25.647601 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:26.767703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:26.767677 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:26.865298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:26.865221 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca\") pod \"977b3c0a-ac76-43c9-867a-b8215c91fd36\" (UID: \"977b3c0a-ac76-43c9-867a-b8215c91fd36\") " Jul 17 06:13:26.865610 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:26.865590 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "977b3c0a-ac76-43c9-867a-b8215c91fd36" (UID: "977b3c0a-ac76-43c9-867a-b8215c91fd36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:13:26.966323 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:26.966224 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977b3c0a-ac76-43c9-867a-b8215c91fd36-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:13:27.637961 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:27.637920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" event={"ID":"977b3c0a-ac76-43c9-867a-b8215c91fd36","Type":"ContainerDied","Data":"52f411b844c064984219fe86da6e4b2e55bb1eb2d624a905a5b6d600c048dc57"} Jul 17 06:13:27.638143 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:27.637982 2574 scope.go:117] "RemoveContainer" containerID="1c60e035c6e02cfc8de1b1656bcf075bf72f460c1222f8f3c081265ab4a7e028" Jul 17 06:13:27.638143 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:27.637941 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls" Jul 17 06:13:27.655605 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:27.655580 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:27.660766 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:27.660742 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-d2xls"] Jul 17 06:13:29.282830 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:13:29.282796 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" path="/var/lib/kubelet/pods/977b3c0a-ac76-43c9-867a-b8215c91fd36/volumes" Jul 17 06:14:05.259190 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259112 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259570 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259584 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259593 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259599 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259657 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259667 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.259680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.259673 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:05.262373 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.262358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:05.264792 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.264769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:14:05.272818 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.272784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:05.303280 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.303230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg\" (UID: \"58e05531-1dab-4a95-b574-3bc428a39e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:05.403802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.403769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg\" (UID: \"58e05531-1dab-4a95-b574-3bc428a39e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:05.404378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.404360 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg\" (UID: \"58e05531-1dab-4a95-b574-3bc428a39e3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:05.577962 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.577889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:05.713381 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.713356 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:05.714715 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:14:05.714687 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e05531_1dab_4a95_b574_3bc428a39e3c.slice/crio-09b45bca92b7e539e5077e04dc9d20fd3db781915e34ce487416e798aba182c8 WatchSource:0}: Error finding container 09b45bca92b7e539e5077e04dc9d20fd3db781915e34ce487416e798aba182c8: Status 404 returned error can't find the container with id 09b45bca92b7e539e5077e04dc9d20fd3db781915e34ce487416e798aba182c8 Jul 17 06:14:05.842996 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:05.842928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" event={"ID":"58e05531-1dab-4a95-b574-3bc428a39e3c","Type":"ContainerStarted","Data":"09b45bca92b7e539e5077e04dc9d20fd3db781915e34ce487416e798aba182c8"} Jul 17 06:14:06.850011 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:06.849982 2574 generic.go:356] "Generic (PLEG): container finished" podID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerID="d5fdff813ca9f1ffd296a0d51bf274718d8e38fabf8fc675f8b7b4b25ff19c34" exitCode=7 Jul 17 06:14:06.850424 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:06.850077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" event={"ID":"58e05531-1dab-4a95-b574-3bc428a39e3c","Type":"ContainerDied","Data":"d5fdff813ca9f1ffd296a0d51bf274718d8e38fabf8fc675f8b7b4b25ff19c34"} Jul 17 06:14:06.850424 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:06.850214 2574 scope.go:117] "RemoveContainer" containerID="d5fdff813ca9f1ffd296a0d51bf274718d8e38fabf8fc675f8b7b4b25ff19c34" Jul 17 06:14:07.857007 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:07.856972 2574 generic.go:356] "Generic (PLEG): container finished" podID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerID="c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e" exitCode=7 Jul 17 06:14:07.857468 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:07.857064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" event={"ID":"58e05531-1dab-4a95-b574-3bc428a39e3c","Type":"ContainerDied","Data":"c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e"} Jul 17 06:14:07.857468 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:07.857123 2574 scope.go:117] "RemoveContainer" containerID="d5fdff813ca9f1ffd296a0d51bf274718d8e38fabf8fc675f8b7b4b25ff19c34" Jul 17 06:14:07.857468 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:07.857270 2574 scope.go:117] "RemoveContainer" containerID="c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e" Jul 17 06:14:07.857643 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:14:07.857489 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg_odh-ai-gateway-infra(58e05531-1dab-4a95-b574-3bc428a39e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" Jul 17 06:14:08.864426 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:08.864398 2574 scope.go:117] "RemoveContainer" containerID="c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e" Jul 17 06:14:08.864911 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:14:08.864642 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg_odh-ai-gateway-infra(58e05531-1dab-4a95-b574-3bc428a39e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" Jul 17 06:14:22.276922 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:22.276893 2574 scope.go:117] "RemoveContainer" containerID="c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e" Jul 17 06:14:22.935986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:22.935952 2574 generic.go:356] "Generic (PLEG): container finished" podID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerID="193b03e6748a4745977248944722a7c4b79d9685b171b34d088b416292e0fa5c" exitCode=7 Jul 17 06:14:22.936154 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:22.936026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" event={"ID":"58e05531-1dab-4a95-b574-3bc428a39e3c","Type":"ContainerDied","Data":"193b03e6748a4745977248944722a7c4b79d9685b171b34d088b416292e0fa5c"} Jul 17 06:14:22.936154 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:22.936088 2574 scope.go:117] "RemoveContainer" containerID="c08411254b813dd0e6f6d89c76c19f203e630980d4b8c1751ac87b9d2558e60e" Jul 17 06:14:22.936368 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:22.936355 2574 scope.go:117] "RemoveContainer" containerID="193b03e6748a4745977248944722a7c4b79d9685b171b34d088b416292e0fa5c" Jul 17 06:14:22.936648 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:14:22.936626 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg_odh-ai-gateway-infra(58e05531-1dab-4a95-b574-3bc428a39e3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" Jul 17 06:14:23.963807 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:23.963773 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:24.100615 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.100585 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:24.166419 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.166375 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca\") pod \"58e05531-1dab-4a95-b574-3bc428a39e3c\" (UID: \"58e05531-1dab-4a95-b574-3bc428a39e3c\") " Jul 17 06:14:24.166758 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.166725 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58e05531-1dab-4a95-b574-3bc428a39e3c" (UID: "58e05531-1dab-4a95-b574-3bc428a39e3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:14:24.267561 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.267475 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e05531-1dab-4a95-b574-3bc428a39e3c-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:14:24.949157 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.949127 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" Jul 17 06:14:24.949390 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.949130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg" event={"ID":"58e05531-1dab-4a95-b574-3bc428a39e3c","Type":"ContainerDied","Data":"09b45bca92b7e539e5077e04dc9d20fd3db781915e34ce487416e798aba182c8"} Jul 17 06:14:24.949390 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.949278 2574 scope.go:117] "RemoveContainer" containerID="193b03e6748a4745977248944722a7c4b79d9685b171b34d088b416292e0fa5c" Jul 17 06:14:24.970237 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.970214 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:24.976361 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:24.976340 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-p8lvg"] Jul 17 06:14:25.282069 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:25.281986 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" path="/var/lib/kubelet/pods/58e05531-1dab-4a95-b574-3bc428a39e3c/volumes" Jul 17 06:14:55.297198 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297167 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297513 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297524 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297538 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297543 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="977b3c0a-ac76-43c9-867a-b8215c91fd36" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297554 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297559 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297618 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.297699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.297625 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:14:55.300305 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.300289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:14:55.302536 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.302518 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:14:55.316351 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.310712 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:14:55.336985 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.336955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj\" (UID: \"988db5aa-16c1-453e-9e1c-09fa58301529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:14:55.438403 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.438365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj\" (UID: \"988db5aa-16c1-453e-9e1c-09fa58301529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:14:55.438982 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.438962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj\" (UID: \"988db5aa-16c1-453e-9e1c-09fa58301529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:14:55.613333 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.613307 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:14:55.746448 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:55.746420 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:14:55.747787 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:14:55.747754 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988db5aa_16c1_453e_9e1c_09fa58301529.slice/crio-ff34025ae2b94f477494bc1cb2a29159d840c76a1027882da5af79787cec5006 WatchSource:0}: Error finding container ff34025ae2b94f477494bc1cb2a29159d840c76a1027882da5af79787cec5006: Status 404 returned error can't find the container with id ff34025ae2b94f477494bc1cb2a29159d840c76a1027882da5af79787cec5006 Jul 17 06:14:56.118556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:56.118525 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerStarted","Data":"ec2d7e3aaa8b71cfa3ac27d94db19fe4b823687873d522440a59ad84bfc14654"} Jul 17 06:14:56.118677 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:56.118577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerStarted","Data":"ff34025ae2b94f477494bc1cb2a29159d840c76a1027882da5af79787cec5006"} Jul 17 06:14:56.132977 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:56.132932 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" podStartSLOduration=1.13291756 podStartE2EDuration="1.13291756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:14:56.13193824 +0000 UTC m=+2023.321318523" watchObservedRunningTime="2026-07-17 06:14:56.13291756 +0000 UTC m=+2023.322297898" Jul 17 06:14:57.124839 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:57.124808 2574 generic.go:356] "Generic (PLEG): container finished" podID="988db5aa-16c1-453e-9e1c-09fa58301529" containerID="ec2d7e3aaa8b71cfa3ac27d94db19fe4b823687873d522440a59ad84bfc14654" exitCode=7 Jul 17 06:14:57.125218 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:57.124876 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerDied","Data":"ec2d7e3aaa8b71cfa3ac27d94db19fe4b823687873d522440a59ad84bfc14654"} Jul 17 06:14:57.125218 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:57.125031 2574 scope.go:117] "RemoveContainer" containerID="ec2d7e3aaa8b71cfa3ac27d94db19fe4b823687873d522440a59ad84bfc14654" Jul 17 06:14:58.131779 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:58.131745 2574 generic.go:356] "Generic (PLEG): container finished" podID="988db5aa-16c1-453e-9e1c-09fa58301529" containerID="76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0" exitCode=7 Jul 17 06:14:58.132297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:58.131833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerDied","Data":"76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0"} Jul 17 06:14:58.132297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:58.131891 2574 scope.go:117] "RemoveContainer" containerID="ec2d7e3aaa8b71cfa3ac27d94db19fe4b823687873d522440a59ad84bfc14654" Jul 17 06:14:58.132297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:58.132123 2574 scope.go:117] "RemoveContainer" containerID="76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0" Jul 17 06:14:58.132484 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:14:58.132405 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj_odh-ai-gateway-infra(988db5aa-16c1-453e-9e1c-09fa58301529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" Jul 17 06:14:59.139953 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:14:59.139926 2574 scope.go:117] "RemoveContainer" containerID="76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0" Jul 17 06:14:59.140364 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:14:59.140137 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj_odh-ai-gateway-infra(988db5aa-16c1-453e-9e1c-09fa58301529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" Jul 17 06:15:00.141957 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.139092 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:00.141957 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.139977 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:15:00.141957 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.140008 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:15:00.141957 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.140278 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e05531-1dab-4a95-b574-3bc428a39e3c" containerName="revoke-keys" Jul 17 06:15:00.145827 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.145801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" Jul 17 06:15:00.152290 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.152242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:00.158635 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.158616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" Jul 17 06:15:00.294530 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:00.294488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:00.297477 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:15:00.297448 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode742a83c_930c_46e8_87a8_ecdbcd4bba61.slice/crio-9d0101da3f882e2f683fbcec56b0be2ff65cd15f113018900f04d0e125a7b1cb WatchSource:0}: Error finding container 9d0101da3f882e2f683fbcec56b0be2ff65cd15f113018900f04d0e125a7b1cb: Status 404 returned error can't find the container with id 9d0101da3f882e2f683fbcec56b0be2ff65cd15f113018900f04d0e125a7b1cb Jul 17 06:15:01.158388 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:01.158303 2574 generic.go:356] "Generic (PLEG): container finished" podID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerID="5fc997178d573b9b1aa16d91b54548d4818415397c58028226edc3c4787d84d4" exitCode=7 Jul 17 06:15:01.158844 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:01.158392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" event={"ID":"e742a83c-930c-46e8-87a8-ecdbcd4bba61","Type":"ContainerDied","Data":"5fc997178d573b9b1aa16d91b54548d4818415397c58028226edc3c4787d84d4"} Jul 17 06:15:01.158844 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:01.158443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" event={"ID":"e742a83c-930c-46e8-87a8-ecdbcd4bba61","Type":"ContainerStarted","Data":"9d0101da3f882e2f683fbcec56b0be2ff65cd15f113018900f04d0e125a7b1cb"} Jul 17 06:15:01.158844 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:01.158504 2574 scope.go:117] "RemoveContainer" containerID="5fc997178d573b9b1aa16d91b54548d4818415397c58028226edc3c4787d84d4" Jul 17 06:15:02.165178 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:02.165085 2574 generic.go:356] "Generic (PLEG): container finished" podID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerID="7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030" exitCode=7 Jul 17 06:15:02.165178 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:02.165159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" event={"ID":"e742a83c-930c-46e8-87a8-ecdbcd4bba61","Type":"ContainerDied","Data":"7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030"} Jul 17 06:15:02.165693 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:02.165219 2574 scope.go:117] "RemoveContainer" containerID="5fc997178d573b9b1aa16d91b54548d4818415397c58028226edc3c4787d84d4" Jul 17 06:15:02.165693 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:02.165371 2574 scope.go:117] "RemoveContainer" containerID="7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030" Jul 17 06:15:02.165693 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:15:02.165604 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br_odh-ai-gateway-infra(e742a83c-930c-46e8-87a8-ecdbcd4bba61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" Jul 17 06:15:03.172613 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:03.172584 2574 scope.go:117] "RemoveContainer" containerID="7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030" Jul 17 06:15:03.172977 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:15:03.172799 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br_odh-ai-gateway-infra(e742a83c-930c-46e8-87a8-ecdbcd4bba61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" Jul 17 06:15:14.276900 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:14.276862 2574 scope.go:117] "RemoveContainer" containerID="76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0" Jul 17 06:15:15.240199 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:15.240111 2574 generic.go:356] "Generic (PLEG): container finished" podID="988db5aa-16c1-453e-9e1c-09fa58301529" containerID="e73c8e7712123126741053250a340bb3b38d09dc3c1060ca2985538c45c52900" exitCode=7 Jul 17 06:15:15.240199 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:15.240179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerDied","Data":"e73c8e7712123126741053250a340bb3b38d09dc3c1060ca2985538c45c52900"} Jul 17 06:15:15.240458 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:15.240241 2574 scope.go:117] "RemoveContainer" containerID="76f58b0b954c8f66efdfd1c1015ed15ebe1701abd2a796e3f0b00e268c877cf0" Jul 17 06:15:15.240513 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:15.240502 2574 scope.go:117] "RemoveContainer" containerID="e73c8e7712123126741053250a340bb3b38d09dc3c1060ca2985538c45c52900" Jul 17 06:15:15.240772 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:15:15.240718 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj_odh-ai-gateway-infra(988db5aa-16c1-453e-9e1c-09fa58301529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" Jul 17 06:15:16.265697 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:16.265668 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:15:16.395076 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:16.395055 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:15:16.523697 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:16.523620 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca\") pod \"988db5aa-16c1-453e-9e1c-09fa58301529\" (UID: \"988db5aa-16c1-453e-9e1c-09fa58301529\") " Jul 17 06:15:16.523948 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:16.523925 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "988db5aa-16c1-453e-9e1c-09fa58301529" (UID: "988db5aa-16c1-453e-9e1c-09fa58301529"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:15:16.624782 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:16.624753 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988db5aa-16c1-453e-9e1c-09fa58301529-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:15:17.254641 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:17.254612 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" Jul 17 06:15:17.254822 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:17.254612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj" event={"ID":"988db5aa-16c1-453e-9e1c-09fa58301529","Type":"ContainerDied","Data":"ff34025ae2b94f477494bc1cb2a29159d840c76a1027882da5af79787cec5006"} Jul 17 06:15:17.254822 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:17.254758 2574 scope.go:117] "RemoveContainer" containerID="e73c8e7712123126741053250a340bb3b38d09dc3c1060ca2985538c45c52900" Jul 17 06:15:17.282019 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:17.281991 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:15:17.283836 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:17.283816 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-6wkvj"] Jul 17 06:15:18.276297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:18.276243 2574 scope.go:117] "RemoveContainer" containerID="7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030" Jul 17 06:15:19.267519 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:19.267429 2574 generic.go:356] "Generic (PLEG): container finished" podID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerID="e6b1e1779203a33fc8671630ac3f3a81c2f1c76fec9110f7402fa8dc8afa9921" exitCode=7 Jul 17 06:15:19.267940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:19.267507 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" event={"ID":"e742a83c-930c-46e8-87a8-ecdbcd4bba61","Type":"ContainerDied","Data":"e6b1e1779203a33fc8671630ac3f3a81c2f1c76fec9110f7402fa8dc8afa9921"} Jul 17 06:15:19.267940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:19.267575 2574 scope.go:117] "RemoveContainer" containerID="7c059c72954df66052c22d104a3e1450f429faf3898bef5e6e320f6370f9b030" Jul 17 06:15:19.267940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:19.267783 2574 scope.go:117] "RemoveContainer" containerID="e6b1e1779203a33fc8671630ac3f3a81c2f1c76fec9110f7402fa8dc8afa9921" Jul 17 06:15:19.268095 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:15:19.268049 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br_odh-ai-gateway-infra(e742a83c-930c-46e8-87a8-ecdbcd4bba61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" Jul 17 06:15:19.282976 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:19.282954 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" path="/var/lib/kubelet/pods/988db5aa-16c1-453e-9e1c-09fa58301529/volumes" Jul 17 06:15:20.293385 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:20.293357 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:20.440889 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:20.440868 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" Jul 17 06:15:21.280618 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:21.280582 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" Jul 17 06:15:21.283365 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:21.283333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br" event={"ID":"e742a83c-930c-46e8-87a8-ecdbcd4bba61","Type":"ContainerDied","Data":"9d0101da3f882e2f683fbcec56b0be2ff65cd15f113018900f04d0e125a7b1cb"} Jul 17 06:15:21.283517 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:21.283397 2574 scope.go:117] "RemoveContainer" containerID="e6b1e1779203a33fc8671630ac3f3a81c2f1c76fec9110f7402fa8dc8afa9921" Jul 17 06:15:21.306241 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:21.306212 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:21.309579 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:21.309558 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c567a84-29737815-vb6br"] Jul 17 06:15:23.282148 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:15:23.282117 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" path="/var/lib/kubelet/pods/e742a83c-930c-46e8-87a8-ecdbcd4bba61/volumes" Jul 17 06:16:13.637863 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:13.637750 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:16:13.649593 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:13.649571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:16:33.247386 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247353 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247697 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247707 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247716 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247722 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247728 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247733 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247757 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.247771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247762 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.248054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247821 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.248054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247829 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.248054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247835 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.248054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247841 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:33.248054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.247848 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:33.249915 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.249898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:33.252355 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.252331 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:16:33.258208 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.258183 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:33.282396 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.282371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh\" (UID: \"7a6a750d-0aba-4bda-848c-b9273c38b9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:33.383719 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.383685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh\" (UID: \"7a6a750d-0aba-4bda-848c-b9273c38b9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:33.384400 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.384379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh\" (UID: \"7a6a750d-0aba-4bda-848c-b9273c38b9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:33.561327 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.561204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:33.698940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:33.698913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:33.701479 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:16:33.701455 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6a750d_0aba_4bda_848c_b9273c38b9d1.slice/crio-21fdfbeb9f921fb747180f8d9bbe36384d39c88a3f42bae67275f798ecbcb111 WatchSource:0}: Error finding container 21fdfbeb9f921fb747180f8d9bbe36384d39c88a3f42bae67275f798ecbcb111: Status 404 returned error can't find the container with id 21fdfbeb9f921fb747180f8d9bbe36384d39c88a3f42bae67275f798ecbcb111 Jul 17 06:16:34.671070 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:34.671031 2574 generic.go:356] "Generic (PLEG): container finished" podID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerID="26f8130e8622d007a69cbaa691b3b2341c43abbf9976a8bd2df23a81e5ab78b9" exitCode=7 Jul 17 06:16:34.671478 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:34.671103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerDied","Data":"26f8130e8622d007a69cbaa691b3b2341c43abbf9976a8bd2df23a81e5ab78b9"} Jul 17 06:16:34.671478 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:34.671147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerStarted","Data":"21fdfbeb9f921fb747180f8d9bbe36384d39c88a3f42bae67275f798ecbcb111"} Jul 17 06:16:34.671478 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:34.671348 2574 scope.go:117] "RemoveContainer" containerID="26f8130e8622d007a69cbaa691b3b2341c43abbf9976a8bd2df23a81e5ab78b9" Jul 17 06:16:35.679299 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:35.679267 2574 generic.go:356] "Generic (PLEG): container finished" podID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerID="29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3" exitCode=7 Jul 17 06:16:35.679700 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:35.679367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerDied","Data":"29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3"} Jul 17 06:16:35.679700 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:35.679433 2574 scope.go:117] "RemoveContainer" containerID="26f8130e8622d007a69cbaa691b3b2341c43abbf9976a8bd2df23a81e5ab78b9" Jul 17 06:16:35.679700 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:35.679593 2574 scope.go:117] "RemoveContainer" containerID="29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3" Jul 17 06:16:35.679878 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:16:35.679844 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh_odh-ai-gateway-infra(7a6a750d-0aba-4bda-848c-b9273c38b9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" Jul 17 06:16:36.687199 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:36.687169 2574 scope.go:117] "RemoveContainer" containerID="29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3" Jul 17 06:16:36.687595 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:16:36.687374 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh_odh-ai-gateway-infra(7a6a750d-0aba-4bda-848c-b9273c38b9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" Jul 17 06:16:51.277104 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.277067 2574 scope.go:117] "RemoveContainer" containerID="29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3" Jul 17 06:16:51.721213 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721177 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:16:51.721634 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721618 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:51.721680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:51.721680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721650 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:51.721680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721657 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e742a83c-930c-46e8-87a8-ecdbcd4bba61" containerName="cleanup" Jul 17 06:16:51.721773 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.721723 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="988db5aa-16c1-453e-9e1c-09fa58301529" containerName="revoke-keys" Jul 17 06:16:51.724851 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.724835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:16:51.731923 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.731882 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:16:51.769138 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.769107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerStarted","Data":"26474168be484b1c95c96b8c6412f97a83641f99463e425394f39618284db837"} Jul 17 06:16:51.784766 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.784714 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" podStartSLOduration=18.784698134 podStartE2EDuration="18.784698134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:16:51.783399541 +0000 UTC m=+2138.972779824" watchObservedRunningTime="2026-07-17 06:16:51.784698134 +0000 UTC m=+2138.974078441" Jul 17 06:16:51.844079 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.844044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-t7rld\" (UID: \"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:16:51.945699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.945669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-t7rld\" (UID: \"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:16:51.946321 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:51.946298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-t7rld\" (UID: \"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:16:52.038288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.038180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:16:52.178430 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.178399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:16:52.180202 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:16:52.180172 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17a3c23_a7dd_4c32_aabb_bbea6ddb19e7.slice/crio-9724748b6937f2a09c14cb2d59c941520e64a9d7aa0ad4a899d349cd07ee6e53 WatchSource:0}: Error finding container 9724748b6937f2a09c14cb2d59c941520e64a9d7aa0ad4a899d349cd07ee6e53: Status 404 returned error can't find the container with id 9724748b6937f2a09c14cb2d59c941520e64a9d7aa0ad4a899d349cd07ee6e53 Jul 17 06:16:52.776606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.776158 2574 generic.go:356] "Generic (PLEG): container finished" podID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerID="26474168be484b1c95c96b8c6412f97a83641f99463e425394f39618284db837" exitCode=6 Jul 17 06:16:52.776606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.776228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerDied","Data":"26474168be484b1c95c96b8c6412f97a83641f99463e425394f39618284db837"} Jul 17 06:16:52.776606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.776318 2574 scope.go:117] "RemoveContainer" containerID="29b6f2b8578ce82f66d0850d1f1e62cc413a66a7b1922d130b4465c68ce3f9d3" Jul 17 06:16:52.777114 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.776666 2574 scope.go:117] "RemoveContainer" containerID="26474168be484b1c95c96b8c6412f97a83641f99463e425394f39618284db837" Jul 17 06:16:52.777114 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:16:52.776911 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh_odh-ai-gateway-infra(7a6a750d-0aba-4bda-848c-b9273c38b9d1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" Jul 17 06:16:52.778133 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.778109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerStarted","Data":"b2b241d01a344a287b118d12d536a6693f8a99a1476ffab3a01554cdae21c993"} Jul 17 06:16:52.778227 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.778153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerStarted","Data":"9724748b6937f2a09c14cb2d59c941520e64a9d7aa0ad4a899d349cd07ee6e53"} Jul 17 06:16:52.794915 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.794892 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:52.823276 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:52.823205 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" podStartSLOduration=1.823188553 podStartE2EDuration="1.823188553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:16:52.822208187 +0000 UTC m=+2140.011588502" watchObservedRunningTime="2026-07-17 06:16:52.823188553 +0000 UTC m=+2140.012568840" Jul 17 06:16:53.786622 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:53.786594 2574 generic.go:356] "Generic (PLEG): container finished" podID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerID="b2b241d01a344a287b118d12d536a6693f8a99a1476ffab3a01554cdae21c993" exitCode=7 Jul 17 06:16:53.786622 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:53.786630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerDied","Data":"b2b241d01a344a287b118d12d536a6693f8a99a1476ffab3a01554cdae21c993"} Jul 17 06:16:53.787115 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:53.786819 2574 scope.go:117] "RemoveContainer" containerID="b2b241d01a344a287b118d12d536a6693f8a99a1476ffab3a01554cdae21c993" Jul 17 06:16:53.929397 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:53.929368 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:54.066933 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.066834 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca\") pod \"7a6a750d-0aba-4bda-848c-b9273c38b9d1\" (UID: \"7a6a750d-0aba-4bda-848c-b9273c38b9d1\") " Jul 17 06:16:54.067185 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.067162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a6a750d-0aba-4bda-848c-b9273c38b9d1" (UID: "7a6a750d-0aba-4bda-848c-b9273c38b9d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:16:54.167524 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.167491 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6a750d-0aba-4bda-848c-b9273c38b9d1-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:16:54.794213 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.794122 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" Jul 17 06:16:54.794213 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.794160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh" event={"ID":"7a6a750d-0aba-4bda-848c-b9273c38b9d1","Type":"ContainerDied","Data":"21fdfbeb9f921fb747180f8d9bbe36384d39c88a3f42bae67275f798ecbcb111"} Jul 17 06:16:54.794764 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.794234 2574 scope.go:117] "RemoveContainer" containerID="26474168be484b1c95c96b8c6412f97a83641f99463e425394f39618284db837" Jul 17 06:16:54.796220 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.796195 2574 generic.go:356] "Generic (PLEG): container finished" podID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerID="60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37" exitCode=7 Jul 17 06:16:54.796352 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.796232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerDied","Data":"60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37"} Jul 17 06:16:54.796582 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.796566 2574 scope.go:117] "RemoveContainer" containerID="60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37" Jul 17 06:16:54.796819 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:16:54.796798 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-t7rld_odh-ai-gateway-infra(d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" Jul 17 06:16:54.803572 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.803555 2574 scope.go:117] "RemoveContainer" containerID="b2b241d01a344a287b118d12d536a6693f8a99a1476ffab3a01554cdae21c993" Jul 17 06:16:54.829206 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.829180 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:54.831221 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:54.831204 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-wpzkh"] Jul 17 06:16:55.281828 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:55.281801 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" path="/var/lib/kubelet/pods/7a6a750d-0aba-4bda-848c-b9273c38b9d1/volumes" Jul 17 06:16:55.804093 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:16:55.804054 2574 scope.go:117] "RemoveContainer" containerID="60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37" Jul 17 06:16:55.804601 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:16:55.804307 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-t7rld_odh-ai-gateway-infra(d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" Jul 17 06:17:08.276627 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:08.276546 2574 scope.go:117] "RemoveContainer" containerID="60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37" Jul 17 06:17:08.878898 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:08.878818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerStarted","Data":"e2dec02978eae003018a5a19225640f0184b48fbd9d63ccd0b47a9dda1fd2a7f"} Jul 17 06:17:09.885935 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:09.885899 2574 generic.go:356] "Generic (PLEG): container finished" podID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerID="e2dec02978eae003018a5a19225640f0184b48fbd9d63ccd0b47a9dda1fd2a7f" exitCode=7 Jul 17 06:17:09.886382 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:09.885976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerDied","Data":"e2dec02978eae003018a5a19225640f0184b48fbd9d63ccd0b47a9dda1fd2a7f"} Jul 17 06:17:09.886382 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:09.886040 2574 scope.go:117] "RemoveContainer" containerID="60c2a25dd2510c1646ea2bd7cc71b11260834d9253c8c9d1503f1d89648daf37" Jul 17 06:17:09.886382 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:09.886293 2574 scope.go:117] "RemoveContainer" containerID="e2dec02978eae003018a5a19225640f0184b48fbd9d63ccd0b47a9dda1fd2a7f" Jul 17 06:17:09.886570 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:17:09.886532 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-t7rld_odh-ai-gateway-infra(d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" Jul 17 06:17:09.910700 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:09.910671 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:17:11.032712 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.032679 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:17:11.117455 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.117423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca\") pod \"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7\" (UID: \"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7\") " Jul 17 06:17:11.117790 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.117769 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" (UID: "d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:17:11.218720 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.218650 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:17:11.899403 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.899373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" Jul 17 06:17:11.899569 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.899423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld" event={"ID":"d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7","Type":"ContainerDied","Data":"9724748b6937f2a09c14cb2d59c941520e64a9d7aa0ad4a899d349cd07ee6e53"} Jul 17 06:17:11.899569 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.899477 2574 scope.go:117] "RemoveContainer" containerID="e2dec02978eae003018a5a19225640f0184b48fbd9d63ccd0b47a9dda1fd2a7f" Jul 17 06:17:11.917633 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.917604 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:17:11.920338 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:11.920316 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-t7rld"] Jul 17 06:17:13.284423 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:17:13.284387 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" path="/var/lib/kubelet/pods/d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7/volumes" Jul 17 06:19:45.545471 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545432 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:19:45.546002 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545944 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.546002 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545965 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.546002 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545977 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.546002 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545986 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.546002 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.545999 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546007 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546026 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546034 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546136 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546148 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546159 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.546288 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.546167 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:19:45.549360 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.549340 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:19:45.551891 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.551870 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:19:45.557022 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.556906 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:19:45.665523 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.665486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw\" (UID: \"3cbf2ca1-5661-4f18-8abf-ccbdf1952426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:19:45.766110 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.766085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw\" (UID: \"3cbf2ca1-5661-4f18-8abf-ccbdf1952426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:19:45.766690 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.766671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw\" (UID: \"3cbf2ca1-5661-4f18-8abf-ccbdf1952426\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:19:45.860488 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:45.860461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:19:46.209399 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:46.209369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:19:46.210525 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:19:46.210499 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbf2ca1_5661_4f18_8abf_ccbdf1952426.slice/crio-a162800fd343ae3f93e9ddfcbc79d1f22cc1a01dc8165e5385ee2bd4bcc652db WatchSource:0}: Error finding container a162800fd343ae3f93e9ddfcbc79d1f22cc1a01dc8165e5385ee2bd4bcc652db: Status 404 returned error can't find the container with id a162800fd343ae3f93e9ddfcbc79d1f22cc1a01dc8165e5385ee2bd4bcc652db Jul 17 06:19:46.700918 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:46.700884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerStarted","Data":"a115bcd1e083991dfb348572fa428b133bb1162f3de45adc08aa05b2d214bf40"} Jul 17 06:19:46.701355 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:46.700942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerStarted","Data":"a162800fd343ae3f93e9ddfcbc79d1f22cc1a01dc8165e5385ee2bd4bcc652db"} Jul 17 06:19:46.724496 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:46.724450 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" podStartSLOduration=1.724435594 podStartE2EDuration="1.724435594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:19:46.721366625 +0000 UTC m=+2313.910746908" watchObservedRunningTime="2026-07-17 06:19:46.724435594 +0000 UTC m=+2313.913815877" Jul 17 06:19:47.707428 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:47.707400 2574 generic.go:356] "Generic (PLEG): container finished" podID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerID="a115bcd1e083991dfb348572fa428b133bb1162f3de45adc08aa05b2d214bf40" exitCode=6 Jul 17 06:19:47.707845 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:47.707443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerDied","Data":"a115bcd1e083991dfb348572fa428b133bb1162f3de45adc08aa05b2d214bf40"} Jul 17 06:19:47.707845 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:47.707654 2574 scope.go:117] "RemoveContainer" containerID="a115bcd1e083991dfb348572fa428b133bb1162f3de45adc08aa05b2d214bf40" Jul 17 06:19:48.715588 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:48.715553 2574 generic.go:356] "Generic (PLEG): container finished" podID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerID="a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe" exitCode=6 Jul 17 06:19:48.716053 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:48.715637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerDied","Data":"a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe"} Jul 17 06:19:48.716053 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:48.715717 2574 scope.go:117] "RemoveContainer" containerID="a115bcd1e083991dfb348572fa428b133bb1162f3de45adc08aa05b2d214bf40" Jul 17 06:19:48.716053 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:48.715962 2574 scope.go:117] "RemoveContainer" containerID="a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe" Jul 17 06:19:48.716224 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:19:48.716201 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw_odh-ai-gateway-infra(3cbf2ca1-5661-4f18-8abf-ccbdf1952426)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" Jul 17 06:19:49.722523 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:19:49.722499 2574 scope.go:117] "RemoveContainer" containerID="a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe" Jul 17 06:19:49.722999 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:19:49.722679 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw_odh-ai-gateway-infra(3cbf2ca1-5661-4f18-8abf-ccbdf1952426)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" Jul 17 06:20:04.276589 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:04.276512 2574 scope.go:117] "RemoveContainer" containerID="a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe" Jul 17 06:20:04.805890 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:04.805850 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerStarted","Data":"ec7030c7f51e73fb7214c379aeb1dbb9ebb3e84fac0b3d172d712a6e80c899ae"} Jul 17 06:20:05.812541 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:05.812509 2574 generic.go:356] "Generic (PLEG): container finished" podID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerID="ec7030c7f51e73fb7214c379aeb1dbb9ebb3e84fac0b3d172d712a6e80c899ae" exitCode=6 Jul 17 06:20:05.813039 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:05.812579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerDied","Data":"ec7030c7f51e73fb7214c379aeb1dbb9ebb3e84fac0b3d172d712a6e80c899ae"} Jul 17 06:20:05.813039 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:05.812658 2574 scope.go:117] "RemoveContainer" containerID="a0aa22b56e7434ee607fe6f97ab38bc29c52cb20de7ef252c658bdde8144fdbe" Jul 17 06:20:05.813039 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:05.812978 2574 scope.go:117] "RemoveContainer" containerID="ec7030c7f51e73fb7214c379aeb1dbb9ebb3e84fac0b3d172d712a6e80c899ae" Jul 17 06:20:05.813210 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:20:05.813188 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw_odh-ai-gateway-infra(3cbf2ca1-5661-4f18-8abf-ccbdf1952426)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" Jul 17 06:20:05.858332 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:05.858297 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:20:06.959157 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:06.959136 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:20:07.053144 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.053117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca\") pod \"3cbf2ca1-5661-4f18-8abf-ccbdf1952426\" (UID: \"3cbf2ca1-5661-4f18-8abf-ccbdf1952426\") " Jul 17 06:20:07.053487 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.053467 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cbf2ca1-5661-4f18-8abf-ccbdf1952426" (UID: "3cbf2ca1-5661-4f18-8abf-ccbdf1952426"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:20:07.154571 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.154542 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cbf2ca1-5661-4f18-8abf-ccbdf1952426-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:20:07.826614 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.826587 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" Jul 17 06:20:07.826614 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.826591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw" event={"ID":"3cbf2ca1-5661-4f18-8abf-ccbdf1952426","Type":"ContainerDied","Data":"a162800fd343ae3f93e9ddfcbc79d1f22cc1a01dc8165e5385ee2bd4bcc652db"} Jul 17 06:20:07.826889 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.826657 2574 scope.go:117] "RemoveContainer" containerID="ec7030c7f51e73fb7214c379aeb1dbb9ebb3e84fac0b3d172d712a6e80c899ae" Jul 17 06:20:07.852486 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.852459 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:20:07.859111 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:07.859090 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-4fzgw"] Jul 17 06:20:09.282094 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:09.282066 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" path="/var/lib/kubelet/pods/3cbf2ca1-5661-4f18-8abf-ccbdf1952426/volumes" Jul 17 06:20:41.939783 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.939750 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940130 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940141 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940157 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940162 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940171 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940177 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940185 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940190 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940242 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940272 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940281 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a3c23-a7dd-4c32-aabb-bbea6ddb19e7" containerName="revoke-keys" Jul 17 06:20:41.940306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.940289 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6a750d-0aba-4bda-848c-b9273c38b9d1" containerName="revoke-keys" Jul 17 06:20:41.942492 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.942476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:41.945392 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.945365 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:20:41.955380 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:41.955357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:42.058415 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:42.058382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-p8fcl\" (UID: \"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:42.159026 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:42.158995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-p8fcl\" (UID: \"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:42.159676 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:42.159649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-p8fcl\" (UID: \"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:42.253286 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:42.253186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:42.397691 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:42.397666 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:42.399487 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:20:42.399451 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd984d_f93a_4370_9fe2_2a2f1b8dfe36.slice/crio-fddc746ff9465b52fc05bd930a7c2babad1896e44c521593b16af08b30ba8626 WatchSource:0}: Error finding container fddc746ff9465b52fc05bd930a7c2babad1896e44c521593b16af08b30ba8626: Status 404 returned error can't find the container with id fddc746ff9465b52fc05bd930a7c2babad1896e44c521593b16af08b30ba8626 Jul 17 06:20:43.008237 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:43.008205 2574 generic.go:356] "Generic (PLEG): container finished" podID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerID="6c564dc99236c1ab1e6ae3f6597f8a0bac4e2d884dbe99b3883e5a6e35ab028b" exitCode=7 Jul 17 06:20:43.008613 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:43.008296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" event={"ID":"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36","Type":"ContainerDied","Data":"6c564dc99236c1ab1e6ae3f6597f8a0bac4e2d884dbe99b3883e5a6e35ab028b"} Jul 17 06:20:43.008613 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:43.008378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" event={"ID":"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36","Type":"ContainerStarted","Data":"fddc746ff9465b52fc05bd930a7c2babad1896e44c521593b16af08b30ba8626"} Jul 17 06:20:43.008613 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:43.008485 2574 scope.go:117] "RemoveContainer" containerID="6c564dc99236c1ab1e6ae3f6597f8a0bac4e2d884dbe99b3883e5a6e35ab028b" Jul 17 06:20:44.015221 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:44.015130 2574 generic.go:356] "Generic (PLEG): container finished" podID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerID="b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273" exitCode=7 Jul 17 06:20:44.015221 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:44.015189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" event={"ID":"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36","Type":"ContainerDied","Data":"b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273"} Jul 17 06:20:44.015755 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:44.015279 2574 scope.go:117] "RemoveContainer" containerID="6c564dc99236c1ab1e6ae3f6597f8a0bac4e2d884dbe99b3883e5a6e35ab028b" Jul 17 06:20:44.015755 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:44.015489 2574 scope.go:117] "RemoveContainer" containerID="b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273" Jul 17 06:20:44.015755 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:20:44.015726 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-p8fcl_odh-ai-gateway-infra(26fd984d-f93a-4370-9fe2-2a2f1b8dfe36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" Jul 17 06:20:45.022640 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:45.022609 2574 scope.go:117] "RemoveContainer" containerID="b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273" Jul 17 06:20:45.023046 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:20:45.022836 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-p8fcl_odh-ai-gateway-infra(26fd984d-f93a-4370-9fe2-2a2f1b8dfe36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" Jul 17 06:20:55.277071 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:55.277039 2574 scope.go:117] "RemoveContainer" containerID="b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273" Jul 17 06:20:56.081603 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:56.081569 2574 generic.go:356] "Generic (PLEG): container finished" podID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerID="1717ab78b04de7f6638b2d5545df5ee554667a326f090cfc86de436e92da72be" exitCode=7 Jul 17 06:20:56.081777 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:56.081622 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" event={"ID":"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36","Type":"ContainerDied","Data":"1717ab78b04de7f6638b2d5545df5ee554667a326f090cfc86de436e92da72be"} Jul 17 06:20:56.081777 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:56.081685 2574 scope.go:117] "RemoveContainer" containerID="b4c27134ae268675b1c02acdcecda3b7860ae90853e099c889e896feaad23273" Jul 17 06:20:56.082013 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:56.081987 2574 scope.go:117] "RemoveContainer" containerID="1717ab78b04de7f6638b2d5545df5ee554667a326f090cfc86de436e92da72be" Jul 17 06:20:56.082238 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:20:56.082221 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-p8fcl_odh-ai-gateway-infra(26fd984d-f93a-4370-9fe2-2a2f1b8dfe36)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" Jul 17 06:20:57.124710 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:57.124677 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:57.276764 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:57.276742 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:57.378349 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:57.378243 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca\") pod \"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36\" (UID: \"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36\") " Jul 17 06:20:57.378633 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:57.378610 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" (UID: "26fd984d-f93a-4370-9fe2-2a2f1b8dfe36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:20:57.479598 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:57.479566 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:20:58.095114 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:58.095075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" event={"ID":"26fd984d-f93a-4370-9fe2-2a2f1b8dfe36","Type":"ContainerDied","Data":"fddc746ff9465b52fc05bd930a7c2babad1896e44c521593b16af08b30ba8626"} Jul 17 06:20:58.095320 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:58.095131 2574 scope.go:117] "RemoveContainer" containerID="1717ab78b04de7f6638b2d5545df5ee554667a326f090cfc86de436e92da72be" Jul 17 06:20:58.095320 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:58.095086 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl" Jul 17 06:20:58.124809 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:58.124785 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:58.130694 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:58.130671 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-p8fcl"] Jul 17 06:20:59.282350 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:20:59.282319 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" path="/var/lib/kubelet/pods/26fd984d-f93a-4370-9fe2-2a2f1b8dfe36/volumes" Jul 17 06:21:13.672518 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:13.672413 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:21:13.693471 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:13.693444 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:21:23.606562 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606527 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606870 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606881 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606894 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606899 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606915 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:21:23.606949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606921 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:21:23.607163 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606978 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cbf2ca1-5661-4f18-8abf-ccbdf1952426" containerName="revoke-keys" Jul 17 06:21:23.607163 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606989 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.607163 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.606996 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.607163 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.607002 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:23.609884 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.609864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:23.612791 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.612769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:21:23.625225 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.625206 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:23.713343 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.713315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh\" (UID: \"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:23.814495 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.814463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh\" (UID: \"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:23.815097 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.815078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh\" (UID: \"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:23.920201 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:23.920127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:24.060939 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:24.060905 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:24.062062 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:21:24.062039 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dfeaa4_9a26_4d3c_b816_2bc95f00d70d.slice/crio-4cf143e408496d4ffa01eba64102115a5f7f2613996a336a193e0ddd2f283f32 WatchSource:0}: Error finding container 4cf143e408496d4ffa01eba64102115a5f7f2613996a336a193e0ddd2f283f32: Status 404 returned error can't find the container with id 4cf143e408496d4ffa01eba64102115a5f7f2613996a336a193e0ddd2f283f32 Jul 17 06:21:24.241887 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:24.241801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" event={"ID":"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d","Type":"ContainerStarted","Data":"4cf143e408496d4ffa01eba64102115a5f7f2613996a336a193e0ddd2f283f32"} Jul 17 06:21:25.248577 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:25.248545 2574 generic.go:356] "Generic (PLEG): container finished" podID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerID="36a3fdf661585cfa91176be5309fc5f80458795a7f1c0d6c5f9175f3e7f49bb4" exitCode=6 Jul 17 06:21:25.248990 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:25.248637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" event={"ID":"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d","Type":"ContainerDied","Data":"36a3fdf661585cfa91176be5309fc5f80458795a7f1c0d6c5f9175f3e7f49bb4"} Jul 17 06:21:25.248990 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:25.248794 2574 scope.go:117] "RemoveContainer" containerID="36a3fdf661585cfa91176be5309fc5f80458795a7f1c0d6c5f9175f3e7f49bb4" Jul 17 06:21:26.255931 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:26.255842 2574 generic.go:356] "Generic (PLEG): container finished" podID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerID="fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8" exitCode=6 Jul 17 06:21:26.255931 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:26.255891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" event={"ID":"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d","Type":"ContainerDied","Data":"fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8"} Jul 17 06:21:26.256429 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:26.255965 2574 scope.go:117] "RemoveContainer" containerID="36a3fdf661585cfa91176be5309fc5f80458795a7f1c0d6c5f9175f3e7f49bb4" Jul 17 06:21:26.256429 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:26.256181 2574 scope.go:117] "RemoveContainer" containerID="fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8" Jul 17 06:21:26.256429 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:26.256407 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh_odh-ai-gateway-infra(29dfeaa4-9a26-4d3c-b816-2bc95f00d70d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" Jul 17 06:21:27.262766 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:27.262735 2574 scope.go:117] "RemoveContainer" containerID="fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8" Jul 17 06:21:27.263154 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:27.262925 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh_odh-ai-gateway-infra(29dfeaa4-9a26-4d3c-b816-2bc95f00d70d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" Jul 17 06:21:31.982330 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:31.980052 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:31.982330 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:31.980857 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:31.982330 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:31.980879 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd984d-f93a-4370-9fe2-2a2f1b8dfe36" containerName="revoke-keys" Jul 17 06:21:31.986438 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:31.986411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:31.989480 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:31.989441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:32.083815 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:32.083780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-psbxl\" (UID: \"e9d69045-0147-4fe9-b21e-85ddf8ca1c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:32.184566 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:32.184536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-psbxl\" (UID: \"e9d69045-0147-4fe9-b21e-85ddf8ca1c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:32.185172 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:32.185155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-psbxl\" (UID: \"e9d69045-0147-4fe9-b21e-85ddf8ca1c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:32.297865 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:32.297796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:32.641162 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:32.641131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:32.642861 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:21:32.642833 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d69045_0147_4fe9_b21e_85ddf8ca1c53.slice/crio-fc81ab3a8a26398b5cffb3666336b55f440e6f79b04a5ee95c5c30b8cc7d2aa1 WatchSource:0}: Error finding container fc81ab3a8a26398b5cffb3666336b55f440e6f79b04a5ee95c5c30b8cc7d2aa1: Status 404 returned error can't find the container with id fc81ab3a8a26398b5cffb3666336b55f440e6f79b04a5ee95c5c30b8cc7d2aa1 Jul 17 06:21:33.295511 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:33.295483 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerID="abeb8da40b46d066292f55420315c5ebcd3d2e10170b668e7ade380a9b6d531b" exitCode=7 Jul 17 06:21:33.295815 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:33.295558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" event={"ID":"e9d69045-0147-4fe9-b21e-85ddf8ca1c53","Type":"ContainerDied","Data":"abeb8da40b46d066292f55420315c5ebcd3d2e10170b668e7ade380a9b6d531b"} Jul 17 06:21:33.295815 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:33.295610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" event={"ID":"e9d69045-0147-4fe9-b21e-85ddf8ca1c53","Type":"ContainerStarted","Data":"fc81ab3a8a26398b5cffb3666336b55f440e6f79b04a5ee95c5c30b8cc7d2aa1"} Jul 17 06:21:33.295815 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:33.295771 2574 scope.go:117] "RemoveContainer" containerID="abeb8da40b46d066292f55420315c5ebcd3d2e10170b668e7ade380a9b6d531b" Jul 17 06:21:34.303013 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:34.302918 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerID="7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db" exitCode=7 Jul 17 06:21:34.303013 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:34.302999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" event={"ID":"e9d69045-0147-4fe9-b21e-85ddf8ca1c53","Type":"ContainerDied","Data":"7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db"} Jul 17 06:21:34.303457 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:34.303065 2574 scope.go:117] "RemoveContainer" containerID="abeb8da40b46d066292f55420315c5ebcd3d2e10170b668e7ade380a9b6d531b" Jul 17 06:21:34.303457 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:34.303168 2574 scope.go:117] "RemoveContainer" containerID="7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db" Jul 17 06:21:34.303457 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:34.303386 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-psbxl_odh-ai-gateway-infra(e9d69045-0147-4fe9-b21e-85ddf8ca1c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" Jul 17 06:21:35.309440 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:35.309409 2574 scope.go:117] "RemoveContainer" containerID="7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db" Jul 17 06:21:35.309830 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:35.309585 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-psbxl_odh-ai-gateway-infra(e9d69045-0147-4fe9-b21e-85ddf8ca1c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" Jul 17 06:21:42.276761 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:42.276730 2574 scope.go:117] "RemoveContainer" containerID="fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8" Jul 17 06:21:43.352566 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:43.352538 2574 generic.go:356] "Generic (PLEG): container finished" podID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerID="441df003a747ee41d30f5c94e4e84d6120dac802672299f3b21d25a7bb98d04c" exitCode=6 Jul 17 06:21:43.353034 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:43.352595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" event={"ID":"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d","Type":"ContainerDied","Data":"441df003a747ee41d30f5c94e4e84d6120dac802672299f3b21d25a7bb98d04c"} Jul 17 06:21:43.353034 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:43.352651 2574 scope.go:117] "RemoveContainer" containerID="fedb5893752f3fdbbd32544d98fdf0726b53a07fcbe245122e0302c7eec009f8" Jul 17 06:21:43.353034 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:43.352989 2574 scope.go:117] "RemoveContainer" containerID="441df003a747ee41d30f5c94e4e84d6120dac802672299f3b21d25a7bb98d04c" Jul 17 06:21:43.353210 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:43.353189 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh_odh-ai-gateway-infra(29dfeaa4-9a26-4d3c-b816-2bc95f00d70d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" Jul 17 06:21:44.389189 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:44.389153 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:44.525746 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:44.525724 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:44.688549 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:44.688466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca\") pod \"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d\" (UID: \"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d\") " Jul 17 06:21:44.688838 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:44.688815 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" (UID: "29dfeaa4-9a26-4d3c-b816-2bc95f00d70d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:21:44.789520 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:44.789493 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:21:45.365618 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:45.365582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" event={"ID":"29dfeaa4-9a26-4d3c-b816-2bc95f00d70d","Type":"ContainerDied","Data":"4cf143e408496d4ffa01eba64102115a5f7f2613996a336a193e0ddd2f283f32"} Jul 17 06:21:45.365618 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:45.365619 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh" Jul 17 06:21:45.365823 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:45.365646 2574 scope.go:117] "RemoveContainer" containerID="441df003a747ee41d30f5c94e4e84d6120dac802672299f3b21d25a7bb98d04c" Jul 17 06:21:45.392265 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:45.392216 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:45.394648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:45.394627 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-tdgdh"] Jul 17 06:21:47.282690 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:47.282659 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" path="/var/lib/kubelet/pods/29dfeaa4-9a26-4d3c-b816-2bc95f00d70d/volumes" Jul 17 06:21:49.277212 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:49.277179 2574 scope.go:117] "RemoveContainer" containerID="7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db" Jul 17 06:21:50.396653 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:50.396619 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerID="caad7330d71e816f8ec5c06778a2c3e081b19873332cffaabe59d23913859e7d" exitCode=7 Jul 17 06:21:50.397052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:50.396701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" event={"ID":"e9d69045-0147-4fe9-b21e-85ddf8ca1c53","Type":"ContainerDied","Data":"caad7330d71e816f8ec5c06778a2c3e081b19873332cffaabe59d23913859e7d"} Jul 17 06:21:50.397052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:50.396792 2574 scope.go:117] "RemoveContainer" containerID="7ade3757d02c8acc4d0e666f8a77d5a5605324163b5d60a7b8329eb221ca87db" Jul 17 06:21:50.397188 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:50.397166 2574 scope.go:117] "RemoveContainer" containerID="caad7330d71e816f8ec5c06778a2c3e081b19873332cffaabe59d23913859e7d" Jul 17 06:21:50.397430 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:21:50.397410 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-psbxl_odh-ai-gateway-infra(e9d69045-0147-4fe9-b21e-85ddf8ca1c53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" Jul 17 06:21:51.428579 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:51.428545 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:51.575969 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:51.575947 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:51.645228 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:51.645201 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca\") pod \"e9d69045-0147-4fe9-b21e-85ddf8ca1c53\" (UID: \"e9d69045-0147-4fe9-b21e-85ddf8ca1c53\") " Jul 17 06:21:51.645572 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:51.645552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9d69045-0147-4fe9-b21e-85ddf8ca1c53" (UID: "e9d69045-0147-4fe9-b21e-85ddf8ca1c53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:21:51.746718 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:51.746638 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d69045-0147-4fe9-b21e-85ddf8ca1c53-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:21:52.413903 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:52.413870 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" Jul 17 06:21:52.413903 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:52.413896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl" event={"ID":"e9d69045-0147-4fe9-b21e-85ddf8ca1c53","Type":"ContainerDied","Data":"fc81ab3a8a26398b5cffb3666336b55f440e6f79b04a5ee95c5c30b8cc7d2aa1"} Jul 17 06:21:52.414149 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:52.413959 2574 scope.go:117] "RemoveContainer" containerID="caad7330d71e816f8ec5c06778a2c3e081b19873332cffaabe59d23913859e7d" Jul 17 06:21:52.434877 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:52.434853 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:52.438556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:52.438532 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-psbxl"] Jul 17 06:21:53.282015 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:21:53.281987 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" path="/var/lib/kubelet/pods/e9d69045-0147-4fe9-b21e-85ddf8ca1c53/volumes" Jul 17 06:23:32.117797 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.117754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118111 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118122 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118131 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118136 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118151 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118157 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118167 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118172 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118177 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118182 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118244 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118278 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118290 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.118308 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.118299 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:23:32.121309 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.121293 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:32.124193 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.124172 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:23:32.132462 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.132441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:32.218810 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.218783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-752xt\" (UID: \"71ee304d-3fd0-48e2-b009-6bba7b3a18ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:32.320171 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.320140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-752xt\" (UID: \"71ee304d-3fd0-48e2-b009-6bba7b3a18ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:32.320797 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.320777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-752xt\" (UID: \"71ee304d-3fd0-48e2-b009-6bba7b3a18ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:32.431494 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.431423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:32.568938 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.568911 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:32.571050 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:23:32.571019 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ee304d_3fd0_48e2_b009_6bba7b3a18ab.slice/crio-5058b94627a40c6c0016fe40b2cb5f194c8f922d1b5f8c2da5ddbd2d75f2e97b WatchSource:0}: Error finding container 5058b94627a40c6c0016fe40b2cb5f194c8f922d1b5f8c2da5ddbd2d75f2e97b: Status 404 returned error can't find the container with id 5058b94627a40c6c0016fe40b2cb5f194c8f922d1b5f8c2da5ddbd2d75f2e97b Jul 17 06:23:32.941603 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.941518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerStarted","Data":"58e10091969f27ecf2b2dae2a15052b3d47dea222d733346d02432fc8b20bd8e"} Jul 17 06:23:32.941603 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.941573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerStarted","Data":"5058b94627a40c6c0016fe40b2cb5f194c8f922d1b5f8c2da5ddbd2d75f2e97b"} Jul 17 06:23:32.958455 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:32.958413 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" podStartSLOduration=0.958397865 podStartE2EDuration="958.397865ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:23:32.955938791 +0000 UTC m=+2540.145319086" watchObservedRunningTime="2026-07-17 06:23:32.958397865 +0000 UTC m=+2540.147778148" Jul 17 06:23:33.948596 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:33.948551 2574 generic.go:356] "Generic (PLEG): container finished" podID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerID="58e10091969f27ecf2b2dae2a15052b3d47dea222d733346d02432fc8b20bd8e" exitCode=6 Jul 17 06:23:33.949050 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:33.948629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerDied","Data":"58e10091969f27ecf2b2dae2a15052b3d47dea222d733346d02432fc8b20bd8e"} Jul 17 06:23:33.949050 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:33.948851 2574 scope.go:117] "RemoveContainer" containerID="58e10091969f27ecf2b2dae2a15052b3d47dea222d733346d02432fc8b20bd8e" Jul 17 06:23:34.955896 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:34.955816 2574 generic.go:356] "Generic (PLEG): container finished" podID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerID="0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615" exitCode=6 Jul 17 06:23:34.955896 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:34.955865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerDied","Data":"0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615"} Jul 17 06:23:34.956362 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:34.955920 2574 scope.go:117] "RemoveContainer" containerID="58e10091969f27ecf2b2dae2a15052b3d47dea222d733346d02432fc8b20bd8e" Jul 17 06:23:34.956362 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:34.956067 2574 scope.go:117] "RemoveContainer" containerID="0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615" Jul 17 06:23:34.956362 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:23:34.956337 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-752xt_odh-ai-gateway-infra(71ee304d-3fd0-48e2-b009-6bba7b3a18ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" Jul 17 06:23:35.963213 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:35.963187 2574 scope.go:117] "RemoveContainer" containerID="0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615" Jul 17 06:23:35.963639 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:23:35.963387 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-752xt_odh-ai-gateway-infra(71ee304d-3fd0-48e2-b009-6bba7b3a18ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" Jul 17 06:23:47.277161 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:47.277126 2574 scope.go:117] "RemoveContainer" containerID="0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615" Jul 17 06:23:48.026342 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:48.026311 2574 generic.go:356] "Generic (PLEG): container finished" podID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerID="eb87d8cf9c1a6aea4c7100cfe3cd91ed34fa3d11717397ce21440fce3dc8598e" exitCode=6 Jul 17 06:23:48.026516 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:48.026386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerDied","Data":"eb87d8cf9c1a6aea4c7100cfe3cd91ed34fa3d11717397ce21440fce3dc8598e"} Jul 17 06:23:48.026516 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:48.026447 2574 scope.go:117] "RemoveContainer" containerID="0ca40a8ba652a2496e49e715adb6c8ffb2d6301b1f0b823a5f71bf88bcbe8615" Jul 17 06:23:48.026772 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:48.026744 2574 scope.go:117] "RemoveContainer" containerID="eb87d8cf9c1a6aea4c7100cfe3cd91ed34fa3d11717397ce21440fce3dc8598e" Jul 17 06:23:48.027044 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:23:48.026978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-752xt_odh-ai-gateway-infra(71ee304d-3fd0-48e2-b009-6bba7b3a18ab)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" Jul 17 06:23:49.059707 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:49.059666 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:49.195369 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:49.195348 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:49.267117 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:49.267090 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca\") pod \"71ee304d-3fd0-48e2-b009-6bba7b3a18ab\" (UID: \"71ee304d-3fd0-48e2-b009-6bba7b3a18ab\") " Jul 17 06:23:49.267454 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:49.267433 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71ee304d-3fd0-48e2-b009-6bba7b3a18ab" (UID: "71ee304d-3fd0-48e2-b009-6bba7b3a18ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:23:49.367902 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:49.367877 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71ee304d-3fd0-48e2-b009-6bba7b3a18ab-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:23:50.040132 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:50.040101 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" Jul 17 06:23:50.040361 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:50.040107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt" event={"ID":"71ee304d-3fd0-48e2-b009-6bba7b3a18ab","Type":"ContainerDied","Data":"5058b94627a40c6c0016fe40b2cb5f194c8f922d1b5f8c2da5ddbd2d75f2e97b"} Jul 17 06:23:50.040361 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:50.040235 2574 scope.go:117] "RemoveContainer" containerID="eb87d8cf9c1a6aea4c7100cfe3cd91ed34fa3d11717397ce21440fce3dc8598e" Jul 17 06:23:50.059327 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:50.059302 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:50.061950 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:50.061931 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-752xt"] Jul 17 06:23:51.283782 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:23:51.283743 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" path="/var/lib/kubelet/pods/71ee304d-3fd0-48e2-b009-6bba7b3a18ab/volumes" Jul 17 06:24:23.739478 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739439 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739796 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739807 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739822 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739827 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739842 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739848 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739915 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29dfeaa4-9a26-4d3c-b816-2bc95f00d70d" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739923 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739928 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.739930 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739935 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d69045-0147-4fe9-b21e-85ddf8ca1c53" containerName="revoke-keys" Jul 17 06:24:23.740328 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.739941 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:24:23.742840 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.742825 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:23.745316 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.745297 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:24:23.761583 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.761544 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:23.868417 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.868377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj\" (UID: \"78f6fe8e-c9d1-4852-8106-a3ca26640910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:23.969779 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.969747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj\" (UID: \"78f6fe8e-c9d1-4852-8106-a3ca26640910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:23.970419 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:23.970402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj\" (UID: \"78f6fe8e-c9d1-4852-8106-a3ca26640910\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:24.057866 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:24.057786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:24.400866 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:24.400841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:24.402795 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:24:24.402767 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f6fe8e_c9d1_4852_8106_a3ca26640910.slice/crio-b8d26bc43d69fb9a5eb6f99a3b6487a32a222400ec53d9b50769cec47770712f WatchSource:0}: Error finding container b8d26bc43d69fb9a5eb6f99a3b6487a32a222400ec53d9b50769cec47770712f: Status 404 returned error can't find the container with id b8d26bc43d69fb9a5eb6f99a3b6487a32a222400ec53d9b50769cec47770712f Jul 17 06:24:25.224100 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:25.224063 2574 generic.go:356] "Generic (PLEG): container finished" podID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerID="b2069a2ffccd4fbb556b12f9e2946828622187771a0ae221eba7d1260d2e425a" exitCode=6 Jul 17 06:24:25.224536 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:25.224158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" event={"ID":"78f6fe8e-c9d1-4852-8106-a3ca26640910","Type":"ContainerDied","Data":"b2069a2ffccd4fbb556b12f9e2946828622187771a0ae221eba7d1260d2e425a"} Jul 17 06:24:25.224536 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:25.224217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" event={"ID":"78f6fe8e-c9d1-4852-8106-a3ca26640910","Type":"ContainerStarted","Data":"b8d26bc43d69fb9a5eb6f99a3b6487a32a222400ec53d9b50769cec47770712f"} Jul 17 06:24:25.224536 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:25.224395 2574 scope.go:117] "RemoveContainer" containerID="b2069a2ffccd4fbb556b12f9e2946828622187771a0ae221eba7d1260d2e425a" Jul 17 06:24:26.231381 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:26.231352 2574 generic.go:356] "Generic (PLEG): container finished" podID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerID="d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269" exitCode=6 Jul 17 06:24:26.231813 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:26.231492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" event={"ID":"78f6fe8e-c9d1-4852-8106-a3ca26640910","Type":"ContainerDied","Data":"d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269"} Jul 17 06:24:26.231813 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:26.231548 2574 scope.go:117] "RemoveContainer" containerID="b2069a2ffccd4fbb556b12f9e2946828622187771a0ae221eba7d1260d2e425a" Jul 17 06:24:26.231813 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:26.231690 2574 scope.go:117] "RemoveContainer" containerID="d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269" Jul 17 06:24:26.231950 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:24:26.231931 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj_odh-ai-gateway-infra(78f6fe8e-c9d1-4852-8106-a3ca26640910)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" Jul 17 06:24:27.238805 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:27.238779 2574 scope.go:117] "RemoveContainer" containerID="d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269" Jul 17 06:24:27.239196 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:24:27.238954 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj_odh-ai-gateway-infra(78f6fe8e-c9d1-4852-8106-a3ca26640910)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" Jul 17 06:24:40.276434 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:40.276403 2574 scope.go:117] "RemoveContainer" containerID="d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269" Jul 17 06:24:41.320310 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:41.320199 2574 generic.go:356] "Generic (PLEG): container finished" podID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerID="c9953a45e7906a1560a606b1623f6dd47fc8b66c4548a20256ad5c8cb3200a06" exitCode=6 Jul 17 06:24:41.320310 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:41.320283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" event={"ID":"78f6fe8e-c9d1-4852-8106-a3ca26640910","Type":"ContainerDied","Data":"c9953a45e7906a1560a606b1623f6dd47fc8b66c4548a20256ad5c8cb3200a06"} Jul 17 06:24:41.320834 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:41.320360 2574 scope.go:117] "RemoveContainer" containerID="d183bd912c511e6e3ce6bd2ab731695432cb651b97e3d4e0e51253b8e297a269" Jul 17 06:24:41.320834 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:41.320670 2574 scope.go:117] "RemoveContainer" containerID="c9953a45e7906a1560a606b1623f6dd47fc8b66c4548a20256ad5c8cb3200a06" Jul 17 06:24:41.320917 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:24:41.320877 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj_odh-ai-gateway-infra(78f6fe8e-c9d1-4852-8106-a3ca26640910)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" Jul 17 06:24:42.359243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:42.359204 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:42.493473 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:42.493449 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:42.633730 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:42.633638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca\") pod \"78f6fe8e-c9d1-4852-8106-a3ca26640910\" (UID: \"78f6fe8e-c9d1-4852-8106-a3ca26640910\") " Jul 17 06:24:42.634001 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:42.633973 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78f6fe8e-c9d1-4852-8106-a3ca26640910" (UID: "78f6fe8e-c9d1-4852-8106-a3ca26640910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:24:42.734321 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:42.734292 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f6fe8e-c9d1-4852-8106-a3ca26640910-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:24:43.333374 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:43.333335 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" event={"ID":"78f6fe8e-c9d1-4852-8106-a3ca26640910","Type":"ContainerDied","Data":"b8d26bc43d69fb9a5eb6f99a3b6487a32a222400ec53d9b50769cec47770712f"} Jul 17 06:24:43.333537 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:43.333383 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj" Jul 17 06:24:43.333537 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:43.333406 2574 scope.go:117] "RemoveContainer" containerID="c9953a45e7906a1560a606b1623f6dd47fc8b66c4548a20256ad5c8cb3200a06" Jul 17 06:24:43.350868 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:43.350847 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:43.355584 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:43.355562 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-7hdcj"] Jul 17 06:24:45.282317 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:24:45.282287 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" path="/var/lib/kubelet/pods/78f6fe8e-c9d1-4852-8106-a3ca26640910/volumes" Jul 17 06:25:13.791000 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.790962 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:13.791522 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791475 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:25:13.791522 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791495 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ee304d-3fd0-48e2-b009-6bba7b3a18ab" containerName="revoke-keys" Jul 17 06:25:13.791648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791530 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.791648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791539 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.791648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791556 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.791648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791565 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.791843 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791657 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.791843 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.791673 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:13.795883 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.795862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:13.798528 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.798463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:25:13.801767 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.801741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:13.894564 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.894533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs\" (UID: \"7245d49e-9160-4a9d-a974-76df74437f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:13.995490 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.995463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs\" (UID: \"7245d49e-9160-4a9d-a974-76df74437f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:13.996062 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:13.996046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs\" (UID: \"7245d49e-9160-4a9d-a974-76df74437f46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:14.108232 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:14.108180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:14.243316 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:14.243286 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:14.244378 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:25:14.244348 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7245d49e_9160_4a9d_a974_76df74437f46.slice/crio-ea3e35b6a2e95832a3b82db7e8a1e3d617ef542cb3663a5bb8de82b83c6de0b6 WatchSource:0}: Error finding container ea3e35b6a2e95832a3b82db7e8a1e3d617ef542cb3663a5bb8de82b83c6de0b6: Status 404 returned error can't find the container with id ea3e35b6a2e95832a3b82db7e8a1e3d617ef542cb3663a5bb8de82b83c6de0b6 Jul 17 06:25:14.497069 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:14.497041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" event={"ID":"7245d49e-9160-4a9d-a974-76df74437f46","Type":"ContainerStarted","Data":"ea3e35b6a2e95832a3b82db7e8a1e3d617ef542cb3663a5bb8de82b83c6de0b6"} Jul 17 06:25:15.503949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:15.503915 2574 generic.go:356] "Generic (PLEG): container finished" podID="7245d49e-9160-4a9d-a974-76df74437f46" containerID="90e0654c5aa001b67b8bed6c18788e3dec0e7ceff393046b2053b41fc7a44ffe" exitCode=6 Jul 17 06:25:15.504378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:15.503961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" event={"ID":"7245d49e-9160-4a9d-a974-76df74437f46","Type":"ContainerDied","Data":"90e0654c5aa001b67b8bed6c18788e3dec0e7ceff393046b2053b41fc7a44ffe"} Jul 17 06:25:15.504378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:15.504218 2574 scope.go:117] "RemoveContainer" containerID="90e0654c5aa001b67b8bed6c18788e3dec0e7ceff393046b2053b41fc7a44ffe" Jul 17 06:25:16.510851 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:16.510764 2574 generic.go:356] "Generic (PLEG): container finished" podID="7245d49e-9160-4a9d-a974-76df74437f46" containerID="34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba" exitCode=6 Jul 17 06:25:16.511243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:16.510858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" event={"ID":"7245d49e-9160-4a9d-a974-76df74437f46","Type":"ContainerDied","Data":"34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba"} Jul 17 06:25:16.511243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:16.510925 2574 scope.go:117] "RemoveContainer" containerID="90e0654c5aa001b67b8bed6c18788e3dec0e7ceff393046b2053b41fc7a44ffe" Jul 17 06:25:16.511243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:16.511016 2574 scope.go:117] "RemoveContainer" containerID="34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba" Jul 17 06:25:16.511503 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:16.511299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs_odh-ai-gateway-infra(7245d49e-9160-4a9d-a974-76df74437f46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" podUID="7245d49e-9160-4a9d-a974-76df74437f46" Jul 17 06:25:17.518221 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:17.518192 2574 scope.go:117] "RemoveContainer" containerID="34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba" Jul 17 06:25:17.518625 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:17.518399 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs_odh-ai-gateway-infra(7245d49e-9160-4a9d-a974-76df74437f46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" podUID="7245d49e-9160-4a9d-a974-76df74437f46" Jul 17 06:25:22.223991 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.223958 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:22.224440 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.224338 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:22.224440 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.224349 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:22.224440 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.224414 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f6fe8e-c9d1-4852-8106-a3ca26640910" containerName="revoke-keys" Jul 17 06:25:22.227271 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.227239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:22.234579 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.234507 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:22.264356 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.264322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mksr4\" (UID: \"479b3fd3-2690-413c-abec-6cc122e8b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:22.365146 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.365098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mksr4\" (UID: \"479b3fd3-2690-413c-abec-6cc122e8b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:22.365769 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.365750 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-mksr4\" (UID: \"479b3fd3-2690-413c-abec-6cc122e8b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:22.538948 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.538859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:22.880149 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:22.880126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:22.881958 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:25:22.881930 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b3fd3_2690_413c_abec_6cc122e8b0d2.slice/crio-6e667f324d4ad228925eb265f94dcd64eb4421ee23f4e131ed521de38936d9a6 WatchSource:0}: Error finding container 6e667f324d4ad228925eb265f94dcd64eb4421ee23f4e131ed521de38936d9a6: Status 404 returned error can't find the container with id 6e667f324d4ad228925eb265f94dcd64eb4421ee23f4e131ed521de38936d9a6 Jul 17 06:25:23.550092 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:23.550058 2574 generic.go:356] "Generic (PLEG): container finished" podID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerID="900b1f2cfae95f3468d6b032509b9e558e30e273fdd6269d41e68582bdcd80c2" exitCode=6 Jul 17 06:25:23.550535 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:23.550141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerDied","Data":"900b1f2cfae95f3468d6b032509b9e558e30e273fdd6269d41e68582bdcd80c2"} Jul 17 06:25:23.550535 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:23.550232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerStarted","Data":"6e667f324d4ad228925eb265f94dcd64eb4421ee23f4e131ed521de38936d9a6"} Jul 17 06:25:23.550535 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:23.550374 2574 scope.go:117] "RemoveContainer" containerID="900b1f2cfae95f3468d6b032509b9e558e30e273fdd6269d41e68582bdcd80c2" Jul 17 06:25:24.557568 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:24.557225 2574 generic.go:356] "Generic (PLEG): container finished" podID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerID="1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d" exitCode=6 Jul 17 06:25:24.557568 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:24.557314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerDied","Data":"1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d"} Jul 17 06:25:24.557568 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:24.557389 2574 scope.go:117] "RemoveContainer" containerID="900b1f2cfae95f3468d6b032509b9e558e30e273fdd6269d41e68582bdcd80c2" Jul 17 06:25:24.558087 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:24.557570 2574 scope.go:117] "RemoveContainer" containerID="1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d" Jul 17 06:25:24.558087 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:24.557842 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mksr4_odh-ai-gateway-infra(479b3fd3-2690-413c-abec-6cc122e8b0d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" Jul 17 06:25:25.563916 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:25.563891 2574 scope.go:117] "RemoveContainer" containerID="1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d" Jul 17 06:25:25.564297 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:25.564089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mksr4_odh-ai-gateway-infra(479b3fd3-2690-413c-abec-6cc122e8b0d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" Jul 17 06:25:29.276401 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:29.276373 2574 scope.go:117] "RemoveContainer" containerID="34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba" Jul 17 06:25:30.593593 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:30.593561 2574 generic.go:356] "Generic (PLEG): container finished" podID="7245d49e-9160-4a9d-a974-76df74437f46" containerID="e23971a635605b905c8ba18662f06e53fbe49a6cd463c54ec078c29eee58602a" exitCode=6 Jul 17 06:25:30.594054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:30.593646 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" event={"ID":"7245d49e-9160-4a9d-a974-76df74437f46","Type":"ContainerDied","Data":"e23971a635605b905c8ba18662f06e53fbe49a6cd463c54ec078c29eee58602a"} Jul 17 06:25:30.594054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:30.593739 2574 scope.go:117] "RemoveContainer" containerID="34df6c49edae5e3a7145d41d8fcaa650442f157ce82b795f66d858acb4e0d8ba" Jul 17 06:25:30.594054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:30.594047 2574 scope.go:117] "RemoveContainer" containerID="e23971a635605b905c8ba18662f06e53fbe49a6cd463c54ec078c29eee58602a" Jul 17 06:25:30.594302 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:30.594281 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs_odh-ai-gateway-infra(7245d49e-9160-4a9d-a974-76df74437f46)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" podUID="7245d49e-9160-4a9d-a974-76df74437f46" Jul 17 06:25:31.621981 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:31.621947 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:31.761096 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:31.761075 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:31.845610 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:31.845562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca\") pod \"7245d49e-9160-4a9d-a974-76df74437f46\" (UID: \"7245d49e-9160-4a9d-a974-76df74437f46\") " Jul 17 06:25:31.845947 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:31.845927 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7245d49e-9160-4a9d-a974-76df74437f46" (UID: "7245d49e-9160-4a9d-a974-76df74437f46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:25:31.947098 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:31.947019 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7245d49e-9160-4a9d-a974-76df74437f46-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:25:32.607297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:32.607241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" event={"ID":"7245d49e-9160-4a9d-a974-76df74437f46","Type":"ContainerDied","Data":"ea3e35b6a2e95832a3b82db7e8a1e3d617ef542cb3663a5bb8de82b83c6de0b6"} Jul 17 06:25:32.607497 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:32.607322 2574 scope.go:117] "RemoveContainer" containerID="e23971a635605b905c8ba18662f06e53fbe49a6cd463c54ec078c29eee58602a" Jul 17 06:25:32.607497 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:32.607274 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs" Jul 17 06:25:32.628523 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:32.628492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:32.634699 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:32.634675 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-bzwhs"] Jul 17 06:25:33.281978 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:33.281946 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7245d49e-9160-4a9d-a974-76df74437f46" path="/var/lib/kubelet/pods/7245d49e-9160-4a9d-a974-76df74437f46/volumes" Jul 17 06:25:39.277030 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:39.276991 2574 scope.go:117] "RemoveContainer" containerID="1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d" Jul 17 06:25:39.656960 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:39.656934 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerStarted","Data":"292a5c4e9ade97b19dbdaa9e58542f2f2354f86d54f74bfe0586f3a5240f458e"} Jul 17 06:25:39.677664 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:39.677621 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" podStartSLOduration=17.677605204 podStartE2EDuration="17.677605204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:25:39.675388133 +0000 UTC m=+2666.864768421" watchObservedRunningTime="2026-07-17 06:25:39.677605204 +0000 UTC m=+2666.866985486" Jul 17 06:25:40.664264 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:40.664226 2574 generic.go:356] "Generic (PLEG): container finished" podID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerID="292a5c4e9ade97b19dbdaa9e58542f2f2354f86d54f74bfe0586f3a5240f458e" exitCode=6 Jul 17 06:25:40.664697 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:40.664306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerDied","Data":"292a5c4e9ade97b19dbdaa9e58542f2f2354f86d54f74bfe0586f3a5240f458e"} Jul 17 06:25:40.664697 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:40.664367 2574 scope.go:117] "RemoveContainer" containerID="1199da497a57d507efd8f6f03830ee15d3c557f4eb371aaa2290d1b36a4b978d" Jul 17 06:25:40.664697 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:40.664608 2574 scope.go:117] "RemoveContainer" containerID="292a5c4e9ade97b19dbdaa9e58542f2f2354f86d54f74bfe0586f3a5240f458e" Jul 17 06:25:40.664877 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:25:40.664816 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-mksr4_odh-ai-gateway-infra(479b3fd3-2690-413c-abec-6cc122e8b0d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" Jul 17 06:25:40.686652 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:40.686625 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:41.804680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:41.804648 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:41.931457 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:41.931372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca\") pod \"479b3fd3-2690-413c-abec-6cc122e8b0d2\" (UID: \"479b3fd3-2690-413c-abec-6cc122e8b0d2\") " Jul 17 06:25:41.931727 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:41.931705 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "479b3fd3-2690-413c-abec-6cc122e8b0d2" (UID: "479b3fd3-2690-413c-abec-6cc122e8b0d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:25:42.032192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.032157 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479b3fd3-2690-413c-abec-6cc122e8b0d2-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:25:42.678270 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.678221 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" Jul 17 06:25:42.678270 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.678236 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4" event={"ID":"479b3fd3-2690-413c-abec-6cc122e8b0d2","Type":"ContainerDied","Data":"6e667f324d4ad228925eb265f94dcd64eb4421ee23f4e131ed521de38936d9a6"} Jul 17 06:25:42.678513 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.678313 2574 scope.go:117] "RemoveContainer" containerID="292a5c4e9ade97b19dbdaa9e58542f2f2354f86d54f74bfe0586f3a5240f458e" Jul 17 06:25:42.699281 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.699243 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:42.705441 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:42.705420 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-mksr4"] Jul 17 06:25:43.282011 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:43.281976 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" path="/var/lib/kubelet/pods/479b3fd3-2690-413c-abec-6cc122e8b0d2/volumes" Jul 17 06:25:59.843796 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.843759 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844261 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844269 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844275 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844283 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844289 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844304 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844308 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844366 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844373 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844383 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:25:59.844395 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.844389 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:25:59.847424 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.847408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:25:59.849945 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.849926 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:25:59.856171 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.856148 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:25:59.886738 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.886711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd\" (UID: \"bee9d484-a9c2-407d-8450-6d86113cbb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:25:59.987106 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.987072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd\" (UID: \"bee9d484-a9c2-407d-8450-6d86113cbb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:25:59.987703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:25:59.987685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd\" (UID: \"bee9d484-a9c2-407d-8450-6d86113cbb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:26:00.160040 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:00.159951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:26:00.295446 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:00.295422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:26:00.296666 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:26:00.296638 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee9d484_a9c2_407d_8450_6d86113cbb53.slice/crio-6df6bb6c9497f828edb1931b4ac5e5614ebaeaa0d8e31d79fbe8a0d3cae243a8 WatchSource:0}: Error finding container 6df6bb6c9497f828edb1931b4ac5e5614ebaeaa0d8e31d79fbe8a0d3cae243a8: Status 404 returned error can't find the container with id 6df6bb6c9497f828edb1931b4ac5e5614ebaeaa0d8e31d79fbe8a0d3cae243a8 Jul 17 06:26:00.774626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:00.774518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerStarted","Data":"c0022388509f5177ea36213ee7e790eeb25ca61a51ed8f96c0686b7c0bbeb0d8"} Jul 17 06:26:00.774626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:00.774586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerStarted","Data":"6df6bb6c9497f828edb1931b4ac5e5614ebaeaa0d8e31d79fbe8a0d3cae243a8"} Jul 17 06:26:00.792534 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:00.792491 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" podStartSLOduration=1.792476603 podStartE2EDuration="1.792476603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:26:00.790354623 +0000 UTC m=+2687.979734905" watchObservedRunningTime="2026-07-17 06:26:00.792476603 +0000 UTC m=+2687.981856886" Jul 17 06:26:01.781054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:01.781022 2574 generic.go:356] "Generic (PLEG): container finished" podID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerID="c0022388509f5177ea36213ee7e790eeb25ca61a51ed8f96c0686b7c0bbeb0d8" exitCode=7 Jul 17 06:26:01.781494 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:01.781085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerDied","Data":"c0022388509f5177ea36213ee7e790eeb25ca61a51ed8f96c0686b7c0bbeb0d8"} Jul 17 06:26:01.781494 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:01.781337 2574 scope.go:117] "RemoveContainer" containerID="c0022388509f5177ea36213ee7e790eeb25ca61a51ed8f96c0686b7c0bbeb0d8" Jul 17 06:26:02.795086 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:02.794871 2574 generic.go:356] "Generic (PLEG): container finished" podID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerID="2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478" exitCode=7 Jul 17 06:26:02.795086 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:02.794993 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerDied","Data":"2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478"} Jul 17 06:26:02.795648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:02.795148 2574 scope.go:117] "RemoveContainer" containerID="c0022388509f5177ea36213ee7e790eeb25ca61a51ed8f96c0686b7c0bbeb0d8" Jul 17 06:26:02.795648 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:02.795383 2574 scope.go:117] "RemoveContainer" containerID="2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478" Jul 17 06:26:02.795747 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:26:02.795666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd_odh-ai-gateway-infra(bee9d484-a9c2-407d-8450-6d86113cbb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" Jul 17 06:26:03.802202 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:03.802179 2574 scope.go:117] "RemoveContainer" containerID="2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478" Jul 17 06:26:03.802652 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:26:03.802418 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd_odh-ai-gateway-infra(bee9d484-a9c2-407d-8450-6d86113cbb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" Jul 17 06:26:13.713771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:13.713656 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:26:13.734316 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:13.734296 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:26:14.276934 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:14.276902 2574 scope.go:117] "RemoveContainer" containerID="2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478" Jul 17 06:26:14.867773 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:14.867743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerStarted","Data":"912735da7561e094584b6d30f589dc58da6a7a296594ddc9ee0e2f7e25fe6e94"} Jul 17 06:26:15.874642 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:15.874608 2574 generic.go:356] "Generic (PLEG): container finished" podID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerID="912735da7561e094584b6d30f589dc58da6a7a296594ddc9ee0e2f7e25fe6e94" exitCode=7 Jul 17 06:26:15.875067 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:15.874678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerDied","Data":"912735da7561e094584b6d30f589dc58da6a7a296594ddc9ee0e2f7e25fe6e94"} Jul 17 06:26:15.875067 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:15.874744 2574 scope.go:117] "RemoveContainer" containerID="2d23c523195260f3fda55d948dbe9df09270d020f2f6b43718e9bfc943d58478" Jul 17 06:26:15.875067 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:15.875008 2574 scope.go:117] "RemoveContainer" containerID="912735da7561e094584b6d30f589dc58da6a7a296594ddc9ee0e2f7e25fe6e94" Jul 17 06:26:15.875226 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:26:15.875206 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd_odh-ai-gateway-infra(bee9d484-a9c2-407d-8450-6d86113cbb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" Jul 17 06:26:15.902205 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:15.902178 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:26:17.016620 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.016599 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:26:17.134088 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.134010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca\") pod \"bee9d484-a9c2-407d-8450-6d86113cbb53\" (UID: \"bee9d484-a9c2-407d-8450-6d86113cbb53\") " Jul 17 06:26:17.134410 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.134391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bee9d484-a9c2-407d-8450-6d86113cbb53" (UID: "bee9d484-a9c2-407d-8450-6d86113cbb53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:26:17.235390 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.235357 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee9d484-a9c2-407d-8450-6d86113cbb53-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:26:17.888495 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.888465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" event={"ID":"bee9d484-a9c2-407d-8450-6d86113cbb53","Type":"ContainerDied","Data":"6df6bb6c9497f828edb1931b4ac5e5614ebaeaa0d8e31d79fbe8a0d3cae243a8"} Jul 17 06:26:17.888689 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.888503 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd" Jul 17 06:26:17.888689 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.888523 2574 scope.go:117] "RemoveContainer" containerID="912735da7561e094584b6d30f589dc58da6a7a296594ddc9ee0e2f7e25fe6e94" Jul 17 06:26:17.904902 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.904877 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:26:17.911726 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:17.911697 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-6vwtd"] Jul 17 06:26:19.282113 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:26:19.282033 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" path="/var/lib/kubelet/pods/bee9d484-a9c2-407d-8450-6d86113cbb53/volumes" Jul 17 06:28:12.398825 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.398788 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399127 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399137 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399153 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399158 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399164 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399169 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399179 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399184 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399191 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399196 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399271 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399280 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399289 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7245d49e-9160-4a9d-a974-76df74437f46" containerName="revoke-keys" Jul 17 06:28:12.399326 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.399295 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="479b3fd3-2690-413c-abec-6cc122e8b0d2" containerName="revoke-keys" Jul 17 06:28:12.402310 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.402294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:12.404634 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.404613 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:28:12.409470 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.409445 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:12.507734 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.507704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4qvqv\" (UID: \"48ffda5b-7c3d-486b-a068-4c76c6ad5343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:12.608672 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.608640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4qvqv\" (UID: \"48ffda5b-7c3d-486b-a068-4c76c6ad5343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:12.609225 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.609208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-572ab047-4qvqv\" (UID: \"48ffda5b-7c3d-486b-a068-4c76c6ad5343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:12.715343 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.715243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:12.856685 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:12.856662 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:12.858020 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:28:12.857996 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ffda5b_7c3d_486b_a068_4c76c6ad5343.slice/crio-134f13898e94c5bdcb18f68d9bfa622752627e0af4bfeab183df23aa7acb9c95 WatchSource:0}: Error finding container 134f13898e94c5bdcb18f68d9bfa622752627e0af4bfeab183df23aa7acb9c95: Status 404 returned error can't find the container with id 134f13898e94c5bdcb18f68d9bfa622752627e0af4bfeab183df23aa7acb9c95 Jul 17 06:28:13.520455 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:13.520368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerStarted","Data":"4139d4adc159f70dd2b2df9963085d42ab9cf6b9b525c8f1f93cfe0c00436a57"} Jul 17 06:28:13.520455 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:13.520423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerStarted","Data":"134f13898e94c5bdcb18f68d9bfa622752627e0af4bfeab183df23aa7acb9c95"} Jul 17 06:28:13.539753 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:13.539708 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" podStartSLOduration=1.539694471 podStartE2EDuration="1.539694471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:13.537639454 +0000 UTC m=+2820.727019736" watchObservedRunningTime="2026-07-17 06:28:13.539694471 +0000 UTC m=+2820.729074754" Jul 17 06:28:14.526518 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:14.526484 2574 generic.go:356] "Generic (PLEG): container finished" podID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerID="4139d4adc159f70dd2b2df9963085d42ab9cf6b9b525c8f1f93cfe0c00436a57" exitCode=6 Jul 17 06:28:14.526916 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:14.526580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerDied","Data":"4139d4adc159f70dd2b2df9963085d42ab9cf6b9b525c8f1f93cfe0c00436a57"} Jul 17 06:28:14.526916 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:14.526740 2574 scope.go:117] "RemoveContainer" containerID="4139d4adc159f70dd2b2df9963085d42ab9cf6b9b525c8f1f93cfe0c00436a57" Jul 17 06:28:15.533998 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:15.533962 2574 generic.go:356] "Generic (PLEG): container finished" podID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerID="de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc" exitCode=6 Jul 17 06:28:15.534499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:15.534002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerDied","Data":"de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc"} Jul 17 06:28:15.534499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:15.534060 2574 scope.go:117] "RemoveContainer" containerID="4139d4adc159f70dd2b2df9963085d42ab9cf6b9b525c8f1f93cfe0c00436a57" Jul 17 06:28:15.534499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:15.534202 2574 scope.go:117] "RemoveContainer" containerID="de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc" Jul 17 06:28:15.534499 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:15.534481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4qvqv_odh-ai-gateway-infra(48ffda5b-7c3d-486b-a068-4c76c6ad5343)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" Jul 17 06:28:16.091109 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.091079 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:16.091554 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.091540 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee9d484-a9c2-407d-8450-6d86113cbb53" containerName="revoke-keys" Jul 17 06:28:16.095490 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.095474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:16.102809 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.102754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:16.136184 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.136155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7\" (UID: \"90ba494b-5b8c-4785-97b2-f9b2849043f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:16.236643 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.236611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7\" (UID: \"90ba494b-5b8c-4785-97b2-f9b2849043f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:16.237273 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.237230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7\" (UID: \"90ba494b-5b8c-4785-97b2-f9b2849043f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:16.408441 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.408347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:16.540794 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.540764 2574 scope.go:117] "RemoveContainer" containerID="de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc" Jul 17 06:28:16.541167 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:16.541022 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4qvqv_odh-ai-gateway-infra(48ffda5b-7c3d-486b-a068-4c76c6ad5343)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" Jul 17 06:28:16.546671 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:16.546638 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:16.548401 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:28:16.548377 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ba494b_5b8c_4785_97b2_f9b2849043f8.slice/crio-b0adf16f3c7d0f49965d39b752bc499190de83cdd649668812050b232c241101 WatchSource:0}: Error finding container b0adf16f3c7d0f49965d39b752bc499190de83cdd649668812050b232c241101: Status 404 returned error can't find the container with id b0adf16f3c7d0f49965d39b752bc499190de83cdd649668812050b232c241101 Jul 17 06:28:17.547330 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:17.547234 2574 generic.go:356] "Generic (PLEG): container finished" podID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerID="0588b004be516bd2f12bb535d6034e99797b2b0bb96af7eff92503985c93daca" exitCode=6 Jul 17 06:28:17.547330 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:17.547320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerDied","Data":"0588b004be516bd2f12bb535d6034e99797b2b0bb96af7eff92503985c93daca"} Jul 17 06:28:17.547799 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:17.547368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerStarted","Data":"b0adf16f3c7d0f49965d39b752bc499190de83cdd649668812050b232c241101"} Jul 17 06:28:17.547799 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:17.547536 2574 scope.go:117] "RemoveContainer" containerID="0588b004be516bd2f12bb535d6034e99797b2b0bb96af7eff92503985c93daca" Jul 17 06:28:18.554691 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:18.554610 2574 generic.go:356] "Generic (PLEG): container finished" podID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerID="1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe" exitCode=6 Jul 17 06:28:18.554691 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:18.554653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerDied","Data":"1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe"} Jul 17 06:28:18.555172 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:18.554701 2574 scope.go:117] "RemoveContainer" containerID="0588b004be516bd2f12bb535d6034e99797b2b0bb96af7eff92503985c93daca" Jul 17 06:28:18.555172 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:18.554901 2574 scope.go:117] "RemoveContainer" containerID="1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe" Jul 17 06:28:18.555172 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:18.555104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7_odh-ai-gateway-infra(90ba494b-5b8c-4785-97b2-f9b2849043f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" Jul 17 06:28:19.561785 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:19.561754 2574 scope.go:117] "RemoveContainer" containerID="1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe" Jul 17 06:28:19.562186 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:19.561961 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7_odh-ai-gateway-infra(90ba494b-5b8c-4785-97b2-f9b2849043f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" Jul 17 06:28:30.276428 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:30.276396 2574 scope.go:117] "RemoveContainer" containerID="de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc" Jul 17 06:28:30.628850 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:30.628817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerStarted","Data":"73e4bfec5ea0a2582db07cee773beff88723010d83c40bf19d94530683fc74c6"} Jul 17 06:28:31.636053 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:31.636017 2574 generic.go:356] "Generic (PLEG): container finished" podID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerID="73e4bfec5ea0a2582db07cee773beff88723010d83c40bf19d94530683fc74c6" exitCode=6 Jul 17 06:28:31.636561 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:31.636080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerDied","Data":"73e4bfec5ea0a2582db07cee773beff88723010d83c40bf19d94530683fc74c6"} Jul 17 06:28:31.636561 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:31.636139 2574 scope.go:117] "RemoveContainer" containerID="de8d7fa0d62aa5bbb4c027b65db902b1cd20ecda77e4a24edfdd5d28046a41dc" Jul 17 06:28:31.636561 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:31.636428 2574 scope.go:117] "RemoveContainer" containerID="73e4bfec5ea0a2582db07cee773beff88723010d83c40bf19d94530683fc74c6" Jul 17 06:28:31.636733 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:31.636683 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-572ab047-4qvqv_odh-ai-gateway-infra(48ffda5b-7c3d-486b-a068-4c76c6ad5343)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" Jul 17 06:28:31.660037 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:31.660002 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:32.772902 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:32.772876 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:32.879839 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:32.879808 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca\") pod \"48ffda5b-7c3d-486b-a068-4c76c6ad5343\" (UID: \"48ffda5b-7c3d-486b-a068-4c76c6ad5343\") " Jul 17 06:28:32.880173 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:32.880151 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48ffda5b-7c3d-486b-a068-4c76c6ad5343" (UID: "48ffda5b-7c3d-486b-a068-4c76c6ad5343"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:28:32.980557 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:32.980475 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ffda5b-7c3d-486b-a068-4c76c6ad5343-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:28:33.279037 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.278945 2574 scope.go:117] "RemoveContainer" containerID="1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe" Jul 17 06:28:33.649568 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.649536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerStarted","Data":"f2b3f55db56996dfb20bd4dbdb59f1ab85ec3efd4f1c9e79acc49871ad25f2df"} Jul 17 06:28:33.650901 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.650875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" event={"ID":"48ffda5b-7c3d-486b-a068-4c76c6ad5343","Type":"ContainerDied","Data":"134f13898e94c5bdcb18f68d9bfa622752627e0af4bfeab183df23aa7acb9c95"} Jul 17 06:28:33.651061 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.650905 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv" Jul 17 06:28:33.651061 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.650931 2574 scope.go:117] "RemoveContainer" containerID="73e4bfec5ea0a2582db07cee773beff88723010d83c40bf19d94530683fc74c6" Jul 17 06:28:33.668365 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.668327 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" podStartSLOduration=17.668312251 podStartE2EDuration="17.668312251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:28:33.665484801 +0000 UTC m=+2840.854865083" watchObservedRunningTime="2026-07-17 06:28:33.668312251 +0000 UTC m=+2840.857692533" Jul 17 06:28:33.681115 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.681093 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:33.689834 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:33.689813 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-572ab047-4qvqv"] Jul 17 06:28:34.658062 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:34.658025 2574 generic.go:356] "Generic (PLEG): container finished" podID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerID="f2b3f55db56996dfb20bd4dbdb59f1ab85ec3efd4f1c9e79acc49871ad25f2df" exitCode=6 Jul 17 06:28:34.658590 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:34.658106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerDied","Data":"f2b3f55db56996dfb20bd4dbdb59f1ab85ec3efd4f1c9e79acc49871ad25f2df"} Jul 17 06:28:34.658590 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:34.658201 2574 scope.go:117] "RemoveContainer" containerID="1caf7d51b80a19512e8ca9cd420bdc4e6e3f6216c4270300a64fa01c2f8f88fe" Jul 17 06:28:34.658590 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:34.658482 2574 scope.go:117] "RemoveContainer" containerID="f2b3f55db56996dfb20bd4dbdb59f1ab85ec3efd4f1c9e79acc49871ad25f2df" Jul 17 06:28:34.658793 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:28:34.658728 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7_odh-ai-gateway-infra(90ba494b-5b8c-4785-97b2-f9b2849043f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" Jul 17 06:28:34.682995 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:34.682971 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:35.282757 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:35.282724 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" path="/var/lib/kubelet/pods/48ffda5b-7c3d-486b-a068-4c76c6ad5343/volumes" Jul 17 06:28:35.812282 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:35.812260 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:35.907393 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:35.907361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca\") pod \"90ba494b-5b8c-4785-97b2-f9b2849043f8\" (UID: \"90ba494b-5b8c-4785-97b2-f9b2849043f8\") " Jul 17 06:28:35.907687 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:35.907664 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90ba494b-5b8c-4785-97b2-f9b2849043f8" (UID: "90ba494b-5b8c-4785-97b2-f9b2849043f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:28:36.008695 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.008624 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ba494b-5b8c-4785-97b2-f9b2849043f8-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:28:36.672384 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.672347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" event={"ID":"90ba494b-5b8c-4785-97b2-f9b2849043f8","Type":"ContainerDied","Data":"b0adf16f3c7d0f49965d39b752bc499190de83cdd649668812050b232c241101"} Jul 17 06:28:36.672586 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.672414 2574 scope.go:117] "RemoveContainer" containerID="f2b3f55db56996dfb20bd4dbdb59f1ab85ec3efd4f1c9e79acc49871ad25f2df" Jul 17 06:28:36.672586 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.672358 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7" Jul 17 06:28:36.696680 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.696654 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:36.704471 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:36.704447 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-g5rd7"] Jul 17 06:28:37.282626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:28:37.282596 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" path="/var/lib/kubelet/pods/90ba494b-5b8c-4785-97b2-f9b2849043f8/volumes" Jul 17 06:29:09.991800 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.991723 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992154 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992169 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992183 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992188 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992198 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992203 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992211 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992216 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992308 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992317 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.992654 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992323 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992654 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992329 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:29:09.992654 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.992334 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:29:09.995026 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.995011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:09.997372 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.997348 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3b164bbd\"/\"openshift-service-ca.crt\"" Jul 17 06:29:09.998291 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.998270 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3b164bbd\"/\"default-dockercfg-r78tz\"" Jul 17 06:29:09.998392 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.998272 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3b164bbd\"/\"kube-root-ca.crt\"" Jul 17 06:29:09.998392 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:09.998343 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3b164bbd\"/\"e2e-lifecycle-model-3b164bbd-kserve-self-signed-certs\"" Jul 17 06:29:10.006985 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.006913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:29:10.113363 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.113525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.113525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzjf\" (UniqueName: \"kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.113525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.113637 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.113637 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.113551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.214869 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.214841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.214869 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.214872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215073 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.214896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215073 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.214951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215073 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.215013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215073 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.215050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzjf\" (UniqueName: \"kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215339 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.215317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215422 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.215384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.215512 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.215492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.217374 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.217356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.217760 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.217738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.223927 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.223903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzjf\" (UniqueName: \"kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf\") pod \"e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.305343 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.305281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:10.443224 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.443203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:29:10.444765 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:29:10.444739 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeeffa1f_f5a7_45e3_92db_a8fe2e5b76c0.slice/crio-61b0175d6d99d1a03e97693aeb77538d6c6416b7420818eede75614ee3738214 WatchSource:0}: Error finding container 61b0175d6d99d1a03e97693aeb77538d6c6416b7420818eede75614ee3738214: Status 404 returned error can't find the container with id 61b0175d6d99d1a03e97693aeb77538d6c6416b7420818eede75614ee3738214 Jul 17 06:29:10.867437 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.867402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" event={"ID":"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0","Type":"ContainerStarted","Data":"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238"} Jul 17 06:29:10.867437 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.867445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" event={"ID":"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0","Type":"ContainerStarted","Data":"61b0175d6d99d1a03e97693aeb77538d6c6416b7420818eede75614ee3738214"} Jul 17 06:29:10.887731 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:10.887686 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" podStartSLOduration=1.8876722049999999 podStartE2EDuration="1.887672205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:29:10.885179556 +0000 UTC m=+2878.074559841" watchObservedRunningTime="2026-07-17 06:29:10.887672205 +0000 UTC m=+2878.077052487" Jul 17 06:29:20.306416 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:20.306381 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:20.306932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:20.306433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:20.319239 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:20.319208 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:29:20.930964 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:29:20.930929 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:30:06.225965 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:06.225930 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:30:21.246942 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.246885 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" podUID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" containerName="main" containerID="cri-o://04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238" gracePeriod=45 Jul 17 06:30:21.286847 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:30:21.286822 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeeffa1f_f5a7_45e3_92db_a8fe2e5b76c0.slice/crio-04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238.scope\": RecentStats: unable to find data in memory cache]" Jul 17 06:30:21.504882 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.504824 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:30:21.645430 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645443 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645480 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645503 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zzjf\" (UniqueName: \"kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645526 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645556 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs\") pod \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\" (UID: \"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0\") " Jul 17 06:30:21.645888 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645667 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home" (OuterVolumeSpecName: "home") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:30:21.645888 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645759 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:30:21.645888 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645822 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:30:21.646021 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645907 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:21.646021 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645926 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:21.646021 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.645939 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:21.647796 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.647763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm" (OuterVolumeSpecName: "dshm") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:30:21.647796 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.647771 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:30:21.647982 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.647797 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf" (OuterVolumeSpecName: "kube-api-access-2zzjf") pod "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" (UID: "eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0"). InnerVolumeSpecName "kube-api-access-2zzjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:30:21.746895 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.746866 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:21.746895 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.746892 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:21.747066 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:21.746901 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzjf\" (UniqueName: \"kubernetes.io/projected/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0-kube-api-access-2zzjf\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:30:22.243949 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.243919 2574 generic.go:356] "Generic (PLEG): container finished" podID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" containerID="04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238" exitCode=0 Jul 17 06:30:22.244144 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.243966 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" Jul 17 06:30:22.244144 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.244023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" event={"ID":"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0","Type":"ContainerDied","Data":"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238"} Jul 17 06:30:22.244144 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.244067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56" event={"ID":"eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0","Type":"ContainerDied","Data":"61b0175d6d99d1a03e97693aeb77538d6c6416b7420818eede75614ee3738214"} Jul 17 06:30:22.244144 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.244086 2574 scope.go:117] "RemoveContainer" containerID="04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238" Jul 17 06:30:22.253742 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.253486 2574 scope.go:117] "RemoveContainer" containerID="04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238" Jul 17 06:30:22.253965 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:30:22.253757 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238\": container with ID starting with 04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238 not found: ID does not exist" containerID="04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238" Jul 17 06:30:22.253965 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.253779 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238"} err="failed to get container status \"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238\": rpc error: code = NotFound desc = could not find container \"04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238\": container with ID starting with 04c470db9f8f3ed547f01cd177fb5826c5aa1b99341cb51ef1073ea4aa9bb238 not found: ID does not exist" Jul 17 06:30:22.266095 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.266072 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:30:22.270298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:22.270275 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3b164bbd/e2e-lifecycle-model-3b164bbd-kserve-559f856947-7nc56"] Jul 17 06:30:23.282369 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:30:23.282337 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" path="/var/lib/kubelet/pods/eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0/volumes" Jul 17 06:31:13.748934 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:31:13.748819 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:31:13.770732 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:31:13.770713 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:32:20.233714 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.233680 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234063 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234076 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234087 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234092 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ffda5b-7c3d-486b-a068-4c76c6ad5343" containerName="revoke-keys" Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234110 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" containerName="main" Jul 17 06:32:20.234147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234115 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" containerName="main" Jul 17 06:32:20.234378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234173 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ba494b-5b8c-4785-97b2-f9b2849043f8" containerName="revoke-keys" Jul 17 06:32:20.234378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.234186 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeeffa1f-f5a7-45e3-92db-a8fe2e5b76c0" containerName="main" Jul 17 06:32:20.237258 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.237229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:20.241702 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.241680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:32:20.250966 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.250946 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:20.275643 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.275613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-8n89w\" (UID: \"83b7e653-a60a-4c44-b2a5-4a4b2754762b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:20.376971 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.376941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-8n89w\" (UID: \"83b7e653-a60a-4c44-b2a5-4a4b2754762b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:20.377631 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.377608 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51e238b9-8n89w\" (UID: \"83b7e653-a60a-4c44-b2a5-4a4b2754762b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:20.547947 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.547849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:20.690223 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.690195 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:20.691213 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:32:20.691187 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b7e653_a60a_4c44_b2a5_4a4b2754762b.slice/crio-6bdb93cec042f05cb7097ac7bee1840a5d2f2f90e809b1e2ef30bd6fe224bff0 WatchSource:0}: Error finding container 6bdb93cec042f05cb7097ac7bee1840a5d2f2f90e809b1e2ef30bd6fe224bff0: Status 404 returned error can't find the container with id 6bdb93cec042f05cb7097ac7bee1840a5d2f2f90e809b1e2ef30bd6fe224bff0 Jul 17 06:32:20.901708 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:20.901675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" event={"ID":"83b7e653-a60a-4c44-b2a5-4a4b2754762b","Type":"ContainerStarted","Data":"6bdb93cec042f05cb7097ac7bee1840a5d2f2f90e809b1e2ef30bd6fe224bff0"} Jul 17 06:32:21.908931 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:21.908898 2574 generic.go:356] "Generic (PLEG): container finished" podID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerID="8f7def401afa63c825b62687a3cd84cd859061db4fb4dda50bad546fcb31c9ca" exitCode=6 Jul 17 06:32:21.909379 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:21.908943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" event={"ID":"83b7e653-a60a-4c44-b2a5-4a4b2754762b","Type":"ContainerDied","Data":"8f7def401afa63c825b62687a3cd84cd859061db4fb4dda50bad546fcb31c9ca"} Jul 17 06:32:21.909379 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:21.909151 2574 scope.go:117] "RemoveContainer" containerID="8f7def401afa63c825b62687a3cd84cd859061db4fb4dda50bad546fcb31c9ca" Jul 17 06:32:22.916552 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:22.916520 2574 generic.go:356] "Generic (PLEG): container finished" podID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerID="c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422" exitCode=6 Jul 17 06:32:22.916977 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:22.916592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" event={"ID":"83b7e653-a60a-4c44-b2a5-4a4b2754762b","Type":"ContainerDied","Data":"c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422"} Jul 17 06:32:22.916977 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:22.916649 2574 scope.go:117] "RemoveContainer" containerID="8f7def401afa63c825b62687a3cd84cd859061db4fb4dda50bad546fcb31c9ca" Jul 17 06:32:22.916977 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:22.916850 2574 scope.go:117] "RemoveContainer" containerID="c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422" Jul 17 06:32:22.917103 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:32:22.917082 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-8n89w_odh-ai-gateway-infra(83b7e653-a60a-4c44-b2a5-4a4b2754762b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" Jul 17 06:32:23.925074 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:23.925047 2574 scope.go:117] "RemoveContainer" containerID="c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422" Jul 17 06:32:23.925486 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:32:23.925218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-8n89w_odh-ai-gateway-infra(83b7e653-a60a-4c44-b2a5-4a4b2754762b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" Jul 17 06:32:36.276583 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.276551 2574 scope.go:117] "RemoveContainer" containerID="c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422" Jul 17 06:32:36.982945 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.982908 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:32:36.986297 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.986244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:36.989611 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.989591 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e77d3d\"/\"openshift-service-ca.crt\"" Jul 17 06:32:36.989611 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.989597 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e77d3d\"/\"default-dockercfg-6qw4f\"" Jul 17 06:32:36.989794 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.989654 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e77d3d\"/\"kube-root-ca.crt\"" Jul 17 06:32:36.989794 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.989597 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e77d3d\"/\"test-model-e77d3d-kserve-self-signed-certs\"" Jul 17 06:32:36.998053 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.998026 2574 generic.go:356] "Generic (PLEG): container finished" podID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerID="51a56cbdc73855bb1f53ca76439c473698a668d76f41213b69e47757fd2d5943" exitCode=6 Jul 17 06:32:36.998187 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.998118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" event={"ID":"83b7e653-a60a-4c44-b2a5-4a4b2754762b","Type":"ContainerDied","Data":"51a56cbdc73855bb1f53ca76439c473698a668d76f41213b69e47757fd2d5943"} Jul 17 06:32:36.998298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.998194 2574 scope.go:117] "RemoveContainer" containerID="c4e6d0e31362168d6d55d3aa6d69386795936869947040d15c3f46bd3cc06422" Jul 17 06:32:36.998529 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.998513 2574 scope.go:117] "RemoveContainer" containerID="51a56cbdc73855bb1f53ca76439c473698a668d76f41213b69e47757fd2d5943" Jul 17 06:32:36.998766 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:32:36.998744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-51e238b9-8n89w_odh-ai-gateway-infra(83b7e653-a60a-4c44-b2a5-4a4b2754762b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" Jul 17 06:32:36.998766 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:36.998752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:32:37.126740 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.126740 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.126740 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.126969 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.126969 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmd7\" (UniqueName: \"kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.126969 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.126911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228158 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmd7\" (UniqueName: \"kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228779 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.228779 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.228751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.230725 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.230703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.230852 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.230837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.238400 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.238374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmd7\" (UniqueName: \"kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7\") pod \"test-model-e77d3d-kserve-77fcf6854f-nzjtn\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.296849 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.296815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:37.440150 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:37.440123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:32:37.440823 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:32:37.440800 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eaae564_78b4_4c2b_8f13_95f07989a578.slice/crio-ef5d0d3463ecf6a2c561738ace40470b76b1fd38bed20624e3ce0379e640fcc2 WatchSource:0}: Error finding container ef5d0d3463ecf6a2c561738ace40470b76b1fd38bed20624e3ce0379e640fcc2: Status 404 returned error can't find the container with id ef5d0d3463ecf6a2c561738ace40470b76b1fd38bed20624e3ce0379e640fcc2 Jul 17 06:32:38.006980 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.006940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" event={"ID":"0eaae564-78b4-4c2b-8f13-95f07989a578","Type":"ContainerStarted","Data":"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace"} Jul 17 06:32:38.007161 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.006989 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" event={"ID":"0eaae564-78b4-4c2b-8f13-95f07989a578","Type":"ContainerStarted","Data":"ef5d0d3463ecf6a2c561738ace40470b76b1fd38bed20624e3ce0379e640fcc2"} Jul 17 06:32:38.029907 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.029854 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" podStartSLOduration=2.029836438 podStartE2EDuration="2.029836438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:32:38.027351819 +0000 UTC m=+3085.216732104" watchObservedRunningTime="2026-07-17 06:32:38.029836438 +0000 UTC m=+3085.219216721" Jul 17 06:32:38.049931 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.049898 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:38.188035 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.188004 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:38.338520 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.338487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca\") pod \"83b7e653-a60a-4c44-b2a5-4a4b2754762b\" (UID: \"83b7e653-a60a-4c44-b2a5-4a4b2754762b\") " Jul 17 06:32:38.338903 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.338846 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83b7e653-a60a-4c44-b2a5-4a4b2754762b" (UID: "83b7e653-a60a-4c44-b2a5-4a4b2754762b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:32:38.439199 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:38.439169 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b7e653-a60a-4c44-b2a5-4a4b2754762b-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:32:39.013327 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.013288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" event={"ID":"83b7e653-a60a-4c44-b2a5-4a4b2754762b","Type":"ContainerDied","Data":"6bdb93cec042f05cb7097ac7bee1840a5d2f2f90e809b1e2ef30bd6fe224bff0"} Jul 17 06:32:39.013327 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.013323 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w" Jul 17 06:32:39.013596 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.013350 2574 scope.go:117] "RemoveContainer" containerID="51a56cbdc73855bb1f53ca76439c473698a668d76f41213b69e47757fd2d5943" Jul 17 06:32:39.036415 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.036387 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:39.041580 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.041544 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51e238b9-8n89w"] Jul 17 06:32:39.282569 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:39.282491 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" path="/var/lib/kubelet/pods/83b7e653-a60a-4c44-b2a5-4a4b2754762b/volumes" Jul 17 06:32:47.297338 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:47.297309 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:47.297719 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:47.297357 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:47.310129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:47.310108 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:32:48.071878 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:32:48.071844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:33:15.258882 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.258840 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259218 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259235 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259266 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259276 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259293 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259301 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259552 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259408 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259552 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259423 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.259552 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.259432 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b7e653-a60a-4c44-b2a5-4a4b2754762b" containerName="revoke-keys" Jul 17 06:33:15.263673 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.263657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.268316 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.268289 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-142438\"/\"kube-root-ca.crt\"" Jul 17 06:33:15.269412 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.269389 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-142438\"/\"default-dockercfg-mx7mh\"" Jul 17 06:33:15.269475 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.269389 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-142438\"/\"openshift-service-ca.crt\"" Jul 17 06:33:15.269475 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.269431 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-142438\"/\"test-model-142438-kserve-self-signed-certs\"" Jul 17 06:33:15.282953 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.282927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:33:15.369655 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.369856 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.369856 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.369856 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.369856 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxgg\" (UniqueName: \"kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.370014 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.369916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470467 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470626 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxgg\" (UniqueName: \"kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470784 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470784 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.470995 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.471101 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.470984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.471101 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.471073 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.473346 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.473314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.473467 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.473356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.480140 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.480115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxgg\" (UniqueName: \"kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg\") pod \"test-model-142438-kserve-8667f79bfc-v2pk4\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.574457 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.574376 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:15.712445 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:15.712418 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:33:15.714347 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:33:15.714319 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65f5fc8a_2165_4a2e_8856_28aa3202e302.slice/crio-4e66a8ae3d6f8ab6cd031b5f59806f5363946d6bb41de8ce2f05487213ee0be8 WatchSource:0}: Error finding container 4e66a8ae3d6f8ab6cd031b5f59806f5363946d6bb41de8ce2f05487213ee0be8: Status 404 returned error can't find the container with id 4e66a8ae3d6f8ab6cd031b5f59806f5363946d6bb41de8ce2f05487213ee0be8 Jul 17 06:33:16.220634 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:16.220602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" event={"ID":"65f5fc8a-2165-4a2e-8856-28aa3202e302","Type":"ContainerStarted","Data":"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c"} Jul 17 06:33:16.220810 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:16.220642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" event={"ID":"65f5fc8a-2165-4a2e-8856-28aa3202e302","Type":"ContainerStarted","Data":"4e66a8ae3d6f8ab6cd031b5f59806f5363946d6bb41de8ce2f05487213ee0be8"} Jul 17 06:33:16.240969 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:16.240903 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" podStartSLOduration=1.2408885299999999 podStartE2EDuration="1.24088853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:33:16.238558985 +0000 UTC m=+3123.427939269" watchObservedRunningTime="2026-07-17 06:33:16.24088853 +0000 UTC m=+3123.430268833" Jul 17 06:33:25.574884 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:25.574845 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:25.575391 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:25.574902 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:25.587718 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:25.587696 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:26.286499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:26.286471 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:33:45.161070 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:45.161033 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:33:46.051851 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:33:46.051815 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:34:00.181576 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.181516 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" podUID="0eaae564-78b4-4c2b-8f13-95f07989a578" containerName="main" containerID="cri-o://46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace" gracePeriod=45 Jul 17 06:34:00.445589 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.445532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:34:00.458522 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.458498 2574 generic.go:356] "Generic (PLEG): container finished" podID="0eaae564-78b4-4c2b-8f13-95f07989a578" containerID="46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace" exitCode=0 Jul 17 06:34:00.458645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.458539 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" Jul 17 06:34:00.458645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.458582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" event={"ID":"0eaae564-78b4-4c2b-8f13-95f07989a578","Type":"ContainerDied","Data":"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace"} Jul 17 06:34:00.458645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.458624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn" event={"ID":"0eaae564-78b4-4c2b-8f13-95f07989a578","Type":"ContainerDied","Data":"ef5d0d3463ecf6a2c561738ace40470b76b1fd38bed20624e3ce0379e640fcc2"} Jul 17 06:34:00.458645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.458643 2574 scope.go:117] "RemoveContainer" containerID="46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace" Jul 17 06:34:00.468527 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.468504 2574 scope.go:117] "RemoveContainer" containerID="46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace" Jul 17 06:34:00.468862 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:34:00.468807 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace\": container with ID starting with 46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace not found: ID does not exist" containerID="46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace" Jul 17 06:34:00.468862 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.468847 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace"} err="failed to get container status \"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace\": rpc error: code = NotFound desc = could not find container \"46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace\": container with ID starting with 46ab9a6a7f643c11095b1c07700ab523768fe87f7c3b600cb123c68f6adf4ace not found: ID does not exist" Jul 17 06:34:00.571759 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571724 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.571910 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571766 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.571910 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571791 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.571910 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571895 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.572049 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571953 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.572049 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.571982 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxmd7\" (UniqueName: \"kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7\") pod \"0eaae564-78b4-4c2b-8f13-95f07989a578\" (UID: \"0eaae564-78b4-4c2b-8f13-95f07989a578\") " Jul 17 06:34:00.572130 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.572047 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:00.572289 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.572240 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.572439 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.572372 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:00.572656 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.572599 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home" (OuterVolumeSpecName: "home") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:00.574207 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.574177 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:34:00.574207 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.574184 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm" (OuterVolumeSpecName: "dshm") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:00.574461 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.574441 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7" (OuterVolumeSpecName: "kube-api-access-bxmd7") pod "0eaae564-78b4-4c2b-8f13-95f07989a578" (UID: "0eaae564-78b4-4c2b-8f13-95f07989a578"). InnerVolumeSpecName "kube-api-access-bxmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:34:00.672798 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.672761 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.672798 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.672790 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.672798 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.672802 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxmd7\" (UniqueName: \"kubernetes.io/projected/0eaae564-78b4-4c2b-8f13-95f07989a578-kube-api-access-bxmd7\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.673033 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.672817 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eaae564-78b4-4c2b-8f13-95f07989a578-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.673033 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.672830 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eaae564-78b4-4c2b-8f13-95f07989a578-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:00.781349 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.781315 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:34:00.791081 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:00.791055 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e77d3d/test-model-e77d3d-kserve-77fcf6854f-nzjtn"] Jul 17 06:34:01.072832 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.072746 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" podUID="65f5fc8a-2165-4a2e-8856-28aa3202e302" containerName="main" containerID="cri-o://043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c" gracePeriod=45 Jul 17 06:34:01.282317 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.282231 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eaae564-78b4-4c2b-8f13-95f07989a578" path="/var/lib/kubelet/pods/0eaae564-78b4-4c2b-8f13-95f07989a578/volumes" Jul 17 06:34:01.339754 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.339731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:34:01.466060 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.466023 2574 generic.go:356] "Generic (PLEG): container finished" podID="65f5fc8a-2165-4a2e-8856-28aa3202e302" containerID="043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c" exitCode=0 Jul 17 06:34:01.466281 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.466078 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" Jul 17 06:34:01.466281 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.466111 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" event={"ID":"65f5fc8a-2165-4a2e-8856-28aa3202e302","Type":"ContainerDied","Data":"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c"} Jul 17 06:34:01.466281 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.466162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4" event={"ID":"65f5fc8a-2165-4a2e-8856-28aa3202e302","Type":"ContainerDied","Data":"4e66a8ae3d6f8ab6cd031b5f59806f5363946d6bb41de8ce2f05487213ee0be8"} Jul 17 06:34:01.466281 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.466185 2574 scope.go:117] "RemoveContainer" containerID="043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c" Jul 17 06:34:01.475064 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.475044 2574 scope.go:117] "RemoveContainer" containerID="043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c" Jul 17 06:34:01.475375 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:34:01.475356 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c\": container with ID starting with 043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c not found: ID does not exist" containerID="043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c" Jul 17 06:34:01.475444 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.475385 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c"} err="failed to get container status \"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c\": rpc error: code = NotFound desc = could not find container \"043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c\": container with ID starting with 043ba99d3c764daed22d5419228672395bf5ef66504b6b6f374491c6323d106c not found: ID does not exist" Jul 17 06:34:01.480657 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.480724 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.480785 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480725 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.480785 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480766 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.480900 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480793 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcxgg\" (UniqueName: \"kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.480900 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480815 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache\") pod \"65f5fc8a-2165-4a2e-8856-28aa3202e302\" (UID: \"65f5fc8a-2165-4a2e-8856-28aa3202e302\") " Jul 17 06:34:01.481010 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.480925 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:01.481067 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.481017 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home" (OuterVolumeSpecName: "home") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:01.481150 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.481127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache" (OuterVolumeSpecName: "model-cache") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:01.481192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.481133 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.481192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.481187 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.483015 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.482990 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm" (OuterVolumeSpecName: "dshm") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:34:01.483015 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.483003 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:34:01.483138 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.483049 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg" (OuterVolumeSpecName: "kube-api-access-xcxgg") pod "65f5fc8a-2165-4a2e-8856-28aa3202e302" (UID: "65f5fc8a-2165-4a2e-8856-28aa3202e302"). InnerVolumeSpecName "kube-api-access-xcxgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:34:01.581646 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.581626 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.581802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.581648 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65f5fc8a-2165-4a2e-8856-28aa3202e302-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.581802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.581660 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxgg\" (UniqueName: \"kubernetes.io/projected/65f5fc8a-2165-4a2e-8856-28aa3202e302-kube-api-access-xcxgg\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.581802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.581668 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65f5fc8a-2165-4a2e-8856-28aa3202e302-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:01.790650 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.790619 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:34:01.797428 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:01.797402 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-142438/test-model-142438-kserve-8667f79bfc-v2pk4"] Jul 17 06:34:03.283965 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:03.283933 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f5fc8a-2165-4a2e-8856-28aa3202e302" path="/var/lib/kubelet/pods/65f5fc8a-2165-4a2e-8856-28aa3202e302/volumes" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.135668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr"] Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136526 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f5fc8a-2165-4a2e-8856-28aa3202e302" containerName="main" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136548 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f5fc8a-2165-4a2e-8856-28aa3202e302" containerName="main" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136570 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eaae564-78b4-4c2b-8f13-95f07989a578" containerName="main" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136577 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eaae564-78b4-4c2b-8f13-95f07989a578" containerName="main" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136729 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f5fc8a-2165-4a2e-8856-28aa3202e302" containerName="main" Jul 17 06:34:56.138452 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.136743 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eaae564-78b4-4c2b-8f13-95f07989a578" containerName="main" Jul 17 06:34:56.140556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.140530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:56.147414 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.147384 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:34:56.158148 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.158124 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr"] Jul 17 06:34:56.267214 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.267179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr\" (UID: \"2c7e0537-1caa-4da1-96ee-18dfb3c12b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:56.368290 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.368231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr\" (UID: \"2c7e0537-1caa-4da1-96ee-18dfb3c12b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:56.368834 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.368815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr\" (UID: \"2c7e0537-1caa-4da1-96ee-18dfb3c12b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:56.451179 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.451099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:56.601238 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.601212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr"] Jul 17 06:34:56.602643 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:34:56.602619 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7e0537_1caa_4da1_96ee_18dfb3c12b89.slice/crio-f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c WatchSource:0}: Error finding container f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c: Status 404 returned error can't find the container with id f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c Jul 17 06:34:56.762898 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:56.762817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" event={"ID":"2c7e0537-1caa-4da1-96ee-18dfb3c12b89","Type":"ContainerStarted","Data":"f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c"} Jul 17 06:34:57.770302 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:57.770239 2574 generic.go:356] "Generic (PLEG): container finished" podID="2c7e0537-1caa-4da1-96ee-18dfb3c12b89" containerID="029a7bcd3837370d89b5ad97dd85f77f18ca5829cf3f43bf0892dc4b0ccf2b6d" exitCode=0 Jul 17 06:34:57.770710 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:57.770353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" event={"ID":"2c7e0537-1caa-4da1-96ee-18dfb3c12b89","Type":"ContainerDied","Data":"029a7bcd3837370d89b5ad97dd85f77f18ca5829cf3f43bf0892dc4b0ccf2b6d"} Jul 17 06:34:58.904528 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:58.904504 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:58.990881 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:58.990852 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca\") pod \"2c7e0537-1caa-4da1-96ee-18dfb3c12b89\" (UID: \"2c7e0537-1caa-4da1-96ee-18dfb3c12b89\") " Jul 17 06:34:58.991241 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:58.991222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c7e0537-1caa-4da1-96ee-18dfb3c12b89" (UID: "2c7e0537-1caa-4da1-96ee-18dfb3c12b89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:34:59.091844 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:59.091813 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c7e0537-1caa-4da1-96ee-18dfb3c12b89-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:34:59.783102 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:59.783066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" event={"ID":"2c7e0537-1caa-4da1-96ee-18dfb3c12b89","Type":"ContainerDied","Data":"f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c"} Jul 17 06:34:59.783102 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:59.783100 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr" Jul 17 06:34:59.783350 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:34:59.783117 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a80612f4a79417807493d997f33657b869a1057af4761824b6bc938a39805c" Jul 17 06:35:24.424270 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.424216 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:24.424711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.424642 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7e0537-1caa-4da1-96ee-18dfb3c12b89" containerName="revoke-keys" Jul 17 06:35:24.424711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.424655 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7e0537-1caa-4da1-96ee-18dfb3c12b89" containerName="revoke-keys" Jul 17 06:35:24.424785 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.424727 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7e0537-1caa-4da1-96ee-18dfb3c12b89" containerName="revoke-keys" Jul 17 06:35:24.428294 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.428278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:24.431339 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.431322 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:35:24.444531 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.444505 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:24.500340 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.500309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh\" (UID: \"aa61143b-0807-4552-a32e-171e6cc7dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:24.600926 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.600896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh\" (UID: \"aa61143b-0807-4552-a32e-171e6cc7dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:24.601543 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.601526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh\" (UID: \"aa61143b-0807-4552-a32e-171e6cc7dfae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:24.738884 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.738801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:24.884456 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.884426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:24.885565 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:35:24.885542 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa61143b_0807_4552_a32e_171e6cc7dfae.slice/crio-0af21e8ffb61b21e71e66f43a2ef666da801bd25ca9774dfeef5ffc49088f250 WatchSource:0}: Error finding container 0af21e8ffb61b21e71e66f43a2ef666da801bd25ca9774dfeef5ffc49088f250: Status 404 returned error can't find the container with id 0af21e8ffb61b21e71e66f43a2ef666da801bd25ca9774dfeef5ffc49088f250 Jul 17 06:35:24.926942 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:24.926917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" event={"ID":"aa61143b-0807-4552-a32e-171e6cc7dfae","Type":"ContainerStarted","Data":"0af21e8ffb61b21e71e66f43a2ef666da801bd25ca9774dfeef5ffc49088f250"} Jul 17 06:35:25.933702 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:25.933616 2574 generic.go:356] "Generic (PLEG): container finished" podID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerID="602357e62d1d9874c997ca47695ae28b434bd3f02d7faede55f325359651aaca" exitCode=6 Jul 17 06:35:25.933702 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:25.933659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" event={"ID":"aa61143b-0807-4552-a32e-171e6cc7dfae","Type":"ContainerDied","Data":"602357e62d1d9874c997ca47695ae28b434bd3f02d7faede55f325359651aaca"} Jul 17 06:35:25.934106 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:25.933893 2574 scope.go:117] "RemoveContainer" containerID="602357e62d1d9874c997ca47695ae28b434bd3f02d7faede55f325359651aaca" Jul 17 06:35:26.941122 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:26.941088 2574 generic.go:356] "Generic (PLEG): container finished" podID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerID="2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465" exitCode=6 Jul 17 06:35:26.941556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:26.941170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" event={"ID":"aa61143b-0807-4552-a32e-171e6cc7dfae","Type":"ContainerDied","Data":"2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465"} Jul 17 06:35:26.941556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:26.941232 2574 scope.go:117] "RemoveContainer" containerID="602357e62d1d9874c997ca47695ae28b434bd3f02d7faede55f325359651aaca" Jul 17 06:35:26.941556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:26.941487 2574 scope.go:117] "RemoveContainer" containerID="2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465" Jul 17 06:35:26.941704 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:26.941687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh_odh-ai-gateway-infra(aa61143b-0807-4552-a32e-171e6cc7dfae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" Jul 17 06:35:27.948454 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:27.948429 2574 scope.go:117] "RemoveContainer" containerID="2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465" Jul 17 06:35:27.948868 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:27.948620 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh_odh-ai-gateway-infra(aa61143b-0807-4552-a32e-171e6cc7dfae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" Jul 17 06:35:36.534182 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.534152 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:36.539220 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.539196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:36.558474 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.558450 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:36.601702 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.601674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-77vqg\" (UID: \"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:36.702771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.702746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-77vqg\" (UID: \"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:36.703360 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.703341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-77vqg\" (UID: \"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:36.849280 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.849237 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:36.987763 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.987733 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:36.989159 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:35:36.989129 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc27fc6_e5dd_48f4_94ad_507d9d95cac8.slice/crio-fb51c650c71682d379eef4a9f4945e4e8a750bd21215c7b0df98c006cf0233f4 WatchSource:0}: Error finding container fb51c650c71682d379eef4a9f4945e4e8a750bd21215c7b0df98c006cf0233f4: Status 404 returned error can't find the container with id fb51c650c71682d379eef4a9f4945e4e8a750bd21215c7b0df98c006cf0233f4 Jul 17 06:35:36.997485 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:36.997445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" event={"ID":"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8","Type":"ContainerStarted","Data":"fb51c650c71682d379eef4a9f4945e4e8a750bd21215c7b0df98c006cf0233f4"} Jul 17 06:35:38.004219 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:38.004127 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerID="902365bed0c3e819e3c1bd438608133cfa85ad305a58ec2810a10e1433e0bb26" exitCode=6 Jul 17 06:35:38.004219 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:38.004185 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" event={"ID":"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8","Type":"ContainerDied","Data":"902365bed0c3e819e3c1bd438608133cfa85ad305a58ec2810a10e1433e0bb26"} Jul 17 06:35:38.004644 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:38.004453 2574 scope.go:117] "RemoveContainer" containerID="902365bed0c3e819e3c1bd438608133cfa85ad305a58ec2810a10e1433e0bb26" Jul 17 06:35:39.011318 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:39.011281 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerID="fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093" exitCode=6 Jul 17 06:35:39.011759 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:39.011323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" event={"ID":"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8","Type":"ContainerDied","Data":"fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093"} Jul 17 06:35:39.011759 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:39.011396 2574 scope.go:117] "RemoveContainer" containerID="902365bed0c3e819e3c1bd438608133cfa85ad305a58ec2810a10e1433e0bb26" Jul 17 06:35:39.011759 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:39.011567 2574 scope.go:117] "RemoveContainer" containerID="fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093" Jul 17 06:35:39.011879 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:39.011824 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-77vqg_odh-ai-gateway-infra(7dc27fc6-e5dd-48f4-94ad-507d9d95cac8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" Jul 17 06:35:40.024804 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:40.024777 2574 scope.go:117] "RemoveContainer" containerID="fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093" Jul 17 06:35:40.025292 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:40.024997 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-77vqg_odh-ai-gateway-infra(7dc27fc6-e5dd-48f4-94ad-507d9d95cac8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" Jul 17 06:35:42.276689 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:42.276658 2574 scope.go:117] "RemoveContainer" containerID="2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465" Jul 17 06:35:43.042237 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:43.042209 2574 generic.go:356] "Generic (PLEG): container finished" podID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerID="ddb9f2b209c3b3bd75800762e3cb8b9c53f455534ce55d25656331a114e93648" exitCode=6 Jul 17 06:35:43.042419 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:43.042289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" event={"ID":"aa61143b-0807-4552-a32e-171e6cc7dfae","Type":"ContainerDied","Data":"ddb9f2b209c3b3bd75800762e3cb8b9c53f455534ce55d25656331a114e93648"} Jul 17 06:35:43.042419 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:43.042350 2574 scope.go:117] "RemoveContainer" containerID="2ca6245540a16806487a9c18e0a84d1a6a39afb65c9c904e8937c145f5380465" Jul 17 06:35:43.042709 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:43.042688 2574 scope.go:117] "RemoveContainer" containerID="ddb9f2b209c3b3bd75800762e3cb8b9c53f455534ce55d25656331a114e93648" Jul 17 06:35:43.042933 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:43.042913 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh_odh-ai-gateway-infra(aa61143b-0807-4552-a32e-171e6cc7dfae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" Jul 17 06:35:44.067516 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:44.067484 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:44.194064 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:44.194037 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:44.270423 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:44.270392 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca\") pod \"aa61143b-0807-4552-a32e-171e6cc7dfae\" (UID: \"aa61143b-0807-4552-a32e-171e6cc7dfae\") " Jul 17 06:35:44.270734 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:44.270712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa61143b-0807-4552-a32e-171e6cc7dfae" (UID: "aa61143b-0807-4552-a32e-171e6cc7dfae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:44.371441 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:44.371410 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa61143b-0807-4552-a32e-171e6cc7dfae-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:35:45.055709 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.055676 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" Jul 17 06:35:45.055861 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.055713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh" event={"ID":"aa61143b-0807-4552-a32e-171e6cc7dfae","Type":"ContainerDied","Data":"0af21e8ffb61b21e71e66f43a2ef666da801bd25ca9774dfeef5ffc49088f250"} Jul 17 06:35:45.055861 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.055795 2574 scope.go:117] "RemoveContainer" containerID="ddb9f2b209c3b3bd75800762e3cb8b9c53f455534ce55d25656331a114e93648" Jul 17 06:35:45.083533 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.083508 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:45.088119 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.088096 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a9646996-nrdqh"] Jul 17 06:35:45.282243 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:45.282209 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" path="/var/lib/kubelet/pods/aa61143b-0807-4552-a32e-171e6cc7dfae/volumes" Jul 17 06:35:55.276505 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:55.276473 2574 scope.go:117] "RemoveContainer" containerID="fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093" Jul 17 06:35:56.116749 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:56.116714 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerID="c34a12e7f8ceb8a7af5ac82351d9432f8d148e1f831f7794d2a3384b289e659a" exitCode=6 Jul 17 06:35:56.116917 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:56.116786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" event={"ID":"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8","Type":"ContainerDied","Data":"c34a12e7f8ceb8a7af5ac82351d9432f8d148e1f831f7794d2a3384b289e659a"} Jul 17 06:35:56.116917 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:56.116847 2574 scope.go:117] "RemoveContainer" containerID="fb8b6ccff9e62e6cc3c0e1b2485f8dc7b5f4b2edfdb1a41899e50c6d1dc78093" Jul 17 06:35:56.117146 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:56.117127 2574 scope.go:117] "RemoveContainer" containerID="c34a12e7f8ceb8a7af5ac82351d9432f8d148e1f831f7794d2a3384b289e659a" Jul 17 06:35:56.117431 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:35:56.117403 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-77vqg_odh-ai-gateway-infra(7dc27fc6-e5dd-48f4-94ad-507d9d95cac8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" Jul 17 06:35:57.143934 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:57.143902 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:57.282208 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:57.282188 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:57.394359 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:57.394279 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca\") pod \"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8\" (UID: \"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8\") " Jul 17 06:35:57.394674 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:57.394648 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" (UID: "7dc27fc6-e5dd-48f4-94ad-507d9d95cac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:35:57.495821 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:57.495794 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:35:58.129755 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:58.129726 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" Jul 17 06:35:58.129940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:58.129771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg" event={"ID":"7dc27fc6-e5dd-48f4-94ad-507d9d95cac8","Type":"ContainerDied","Data":"fb51c650c71682d379eef4a9f4945e4e8a750bd21215c7b0df98c006cf0233f4"} Jul 17 06:35:58.129940 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:58.129834 2574 scope.go:117] "RemoveContainer" containerID="c34a12e7f8ceb8a7af5ac82351d9432f8d148e1f831f7794d2a3384b289e659a" Jul 17 06:35:58.150349 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:58.150329 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:58.154291 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:58.154269 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-77vqg"] Jul 17 06:35:59.282342 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:35:59.282312 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" path="/var/lib/kubelet/pods/7dc27fc6-e5dd-48f4-94ad-507d9d95cac8/volumes" Jul 17 06:36:13.781735 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:13.781630 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:36:13.806386 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:13.806363 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:36:18.487662 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.487623 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.487971 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.487982 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.487992 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.487998 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488021 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488028 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488034 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488052 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488039 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488100 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488106 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488113 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488120 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488125 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:36:18.488348 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.488133 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:36:18.491226 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.491206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.494609 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.494589 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"kube-root-ca.crt\"" Jul 17 06:36:18.494721 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.494622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"auth-test-model-ff4257-kserve-self-signed-certs\"" Jul 17 06:36:18.495108 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.495087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"openshift-service-ca.crt\"" Jul 17 06:36:18.495188 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.495102 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"default-dockercfg-c44q4\"" Jul 17 06:36:18.501899 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.501880 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:36:18.576477 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.576477 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.576677 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.576677 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwxs\" (UniqueName: \"kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.576745 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.576745 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.576730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677558 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwxs\" (UniqueName: \"kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677866 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.677983 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.677959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.678031 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.678002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.678121 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.678037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.680036 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.680017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.680215 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.680198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.689270 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.689228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwxs\" (UniqueName: \"kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs\") pod \"auth-test-model-ff4257-kserve-67b84d4859-q7pcl\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.800314 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.800232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:18.940840 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:18.940813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:36:18.942372 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:36:18.942347 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd76f6f32_bb53_4a8d_96d1_8f17eec591e1.slice/crio-58590c3664eca7723dab5f59ce9813f73fa370b90544b21da3c7603b45da1f0b WatchSource:0}: Error finding container 58590c3664eca7723dab5f59ce9813f73fa370b90544b21da3c7603b45da1f0b: Status 404 returned error can't find the container with id 58590c3664eca7723dab5f59ce9813f73fa370b90544b21da3c7603b45da1f0b Jul 17 06:36:19.251453 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:19.251418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" event={"ID":"d76f6f32-bb53-4a8d-96d1-8f17eec591e1","Type":"ContainerStarted","Data":"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1"} Jul 17 06:36:19.251453 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:19.251464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" event={"ID":"d76f6f32-bb53-4a8d-96d1-8f17eec591e1","Type":"ContainerStarted","Data":"58590c3664eca7723dab5f59ce9813f73fa370b90544b21da3c7603b45da1f0b"} Jul 17 06:36:19.269952 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:19.269900 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" podStartSLOduration=1.269880999 podStartE2EDuration="1.269880999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:36:19.269079213 +0000 UTC m=+3306.458459497" watchObservedRunningTime="2026-07-17 06:36:19.269880999 +0000 UTC m=+3306.459261283" Jul 17 06:36:28.800818 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:28.800779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:28.801383 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:28.800836 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:28.813309 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:28.813276 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:36:29.315163 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:36:29.315136 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:38:42.013796 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:42.013761 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:38:57.035009 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.034962 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" podUID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" containerName="main" containerID="cri-o://bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1" gracePeriod=45 Jul 17 06:38:57.292425 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.292363 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:38:57.378047 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378017 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpwxs\" (UniqueName: \"kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378197 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378086 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378197 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378110 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378197 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378126 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378197 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378188 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378445 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378223 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm\") pod \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\" (UID: \"d76f6f32-bb53-4a8d-96d1-8f17eec591e1\") " Jul 17 06:38:57.378445 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378364 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home" (OuterVolumeSpecName: "home") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:38:57.378558 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378466 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:38:57.378558 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:38:57.378670 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378613 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:57.378670 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378630 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:57.378670 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.378645 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:57.380463 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.380434 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm" (OuterVolumeSpecName: "dshm") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:38:57.380575 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.380472 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:38:57.380575 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.380542 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs" (OuterVolumeSpecName: "kube-api-access-gpwxs") pod "d76f6f32-bb53-4a8d-96d1-8f17eec591e1" (UID: "d76f6f32-bb53-4a8d-96d1-8f17eec591e1"). InnerVolumeSpecName "kube-api-access-gpwxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:38:57.479525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.479500 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:57.479525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.479523 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwxs\" (UniqueName: \"kubernetes.io/projected/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-kube-api-access-gpwxs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:57.479682 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:57.479533 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d76f6f32-bb53-4a8d-96d1-8f17eec591e1-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:38:58.099644 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.099613 2574 generic.go:356] "Generic (PLEG): container finished" podID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" containerID="bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1" exitCode=0 Jul 17 06:38:58.100083 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.099653 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" Jul 17 06:38:58.100083 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.099686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" event={"ID":"d76f6f32-bb53-4a8d-96d1-8f17eec591e1","Type":"ContainerDied","Data":"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1"} Jul 17 06:38:58.100083 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.099727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl" event={"ID":"d76f6f32-bb53-4a8d-96d1-8f17eec591e1","Type":"ContainerDied","Data":"58590c3664eca7723dab5f59ce9813f73fa370b90544b21da3c7603b45da1f0b"} Jul 17 06:38:58.100083 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.099744 2574 scope.go:117] "RemoveContainer" containerID="bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1" Jul 17 06:38:58.109606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.109590 2574 scope.go:117] "RemoveContainer" containerID="bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1" Jul 17 06:38:58.109848 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:38:58.109829 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1\": container with ID starting with bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1 not found: ID does not exist" containerID="bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1" Jul 17 06:38:58.109941 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.109853 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1"} err="failed to get container status \"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1\": rpc error: code = NotFound desc = could not find container \"bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1\": container with ID starting with bea47d38192d8d0425cedc6162a78e0dc53e5b37564b3301dd9a61b9afcdefd1 not found: ID does not exist" Jul 17 06:38:58.120172 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.120149 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:38:58.124934 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.124913 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/auth-test-model-ff4257-kserve-67b84d4859-q7pcl"] Jul 17 06:38:58.487879 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.487798 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:38:58.488360 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488337 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:38:58.488360 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488362 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc27fc6-e5dd-48f4-94ad-507d9d95cac8" containerName="revoke-keys" Jul 17 06:38:58.488563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488390 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:38:58.488563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488398 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa61143b-0807-4552-a32e-171e6cc7dfae" containerName="revoke-keys" Jul 17 06:38:58.488563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488412 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" containerName="main" Jul 17 06:38:58.488563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488418 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" containerName="main" Jul 17 06:38:58.488563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.488482 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" containerName="main" Jul 17 06:38:58.492721 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.492704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.494908 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.494886 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"kube-root-ca.crt\"" Jul 17 06:38:58.495049 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.495034 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"openshift-service-ca.crt\"" Jul 17 06:38:58.495889 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.495870 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"sub-test-model-ff4257-kserve-self-signed-certs\"" Jul 17 06:38:58.496001 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.495893 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"default-dockercfg-c44q4\"" Jul 17 06:38:58.503377 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.503355 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:38:58.589548 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.589703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.589703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99q2\" (UniqueName: \"kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.589703 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.589868 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.589868 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.589801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690834 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99q2\" (UniqueName: \"kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.690973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.690958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.691213 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.691176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.691355 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.691337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.691355 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.691348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.693236 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.693217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.693599 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.693580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.698851 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.698828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99q2\" (UniqueName: \"kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2\") pod \"sub-test-model-ff4257-kserve-784d7f44b7-vf6pq\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.803688 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.803604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:38:58.963178 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:58.963156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:38:58.965543 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:38:58.965517 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d79c01b_d32b_4e39_bc51_19cec06bb576.slice/crio-c7be31b982a4e295f7741e3535c482a0eeba0b5d95ee8991bfb4c31c51f76a52 WatchSource:0}: Error finding container c7be31b982a4e295f7741e3535c482a0eeba0b5d95ee8991bfb4c31c51f76a52: Status 404 returned error can't find the container with id c7be31b982a4e295f7741e3535c482a0eeba0b5d95ee8991bfb4c31c51f76a52 Jul 17 06:38:59.108224 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:59.108188 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" event={"ID":"3d79c01b-d32b-4e39-bc51-19cec06bb576","Type":"ContainerStarted","Data":"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e"} Jul 17 06:38:59.108224 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:59.108228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" event={"ID":"3d79c01b-d32b-4e39-bc51-19cec06bb576","Type":"ContainerStarted","Data":"c7be31b982a4e295f7741e3535c482a0eeba0b5d95ee8991bfb4c31c51f76a52"} Jul 17 06:38:59.127170 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:59.127117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" podStartSLOduration=1.127101331 podStartE2EDuration="1.127101331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:38:59.126684656 +0000 UTC m=+3466.316064940" watchObservedRunningTime="2026-07-17 06:38:59.127101331 +0000 UTC m=+3466.316481615" Jul 17 06:38:59.282811 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:38:59.282780 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d76f6f32-bb53-4a8d-96d1-8f17eec591e1" path="/var/lib/kubelet/pods/d76f6f32-bb53-4a8d-96d1-8f17eec591e1/volumes" Jul 17 06:39:08.804180 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:39:08.804149 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:39:08.804694 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:39:08.804297 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:39:08.816574 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:39:08.816553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:39:09.172270 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:39:09.172226 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:39:55.975530 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:39:55.975498 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:40:09.874467 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.874434 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:40:09.877963 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.877943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.880571 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.880549 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ff4257\"/\"rate-test-model-ff4257-kserve-self-signed-certs\"" Jul 17 06:40:09.890880 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.890853 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:40:09.923633 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7stx\" (UniqueName: \"kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.923753 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.923753 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.923753 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.923881 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:09.923881 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:09.923849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.024962 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.024919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025120 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025120 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7stx\" (UniqueName: \"kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025120 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025336 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025336 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025472 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025451 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025526 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.025526 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.025520 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.027551 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.027528 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.027846 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.027829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.032869 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.032845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7stx\" (UniqueName: \"kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx\") pod \"rate-test-model-ff4257-kserve-69df4459bc-bdw4p\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.188081 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.187984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:10.329313 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.329287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:40:10.330368 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:40:10.330337 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb221d014_4676_4db7_beb7_8f5aa4a7e8a9.slice/crio-f1a303625c3800c18ce4c234ec4e5b4f4e32ede765c0c6e7f515c33a86f5a1d1 WatchSource:0}: Error finding container f1a303625c3800c18ce4c234ec4e5b4f4e32ede765c0c6e7f515c33a86f5a1d1: Status 404 returned error can't find the container with id f1a303625c3800c18ce4c234ec4e5b4f4e32ede765c0c6e7f515c33a86f5a1d1 Jul 17 06:40:10.485099 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.485019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" event={"ID":"b221d014-4676-4db7-beb7-8f5aa4a7e8a9","Type":"ContainerStarted","Data":"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe"} Jul 17 06:40:10.485099 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.485056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" event={"ID":"b221d014-4676-4db7-beb7-8f5aa4a7e8a9","Type":"ContainerStarted","Data":"f1a303625c3800c18ce4c234ec4e5b4f4e32ede765c0c6e7f515c33a86f5a1d1"} Jul 17 06:40:10.502676 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.502627 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" podStartSLOduration=1.502611737 podStartE2EDuration="1.502611737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 06:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:40:10.501063404 +0000 UTC m=+3537.690443680" watchObservedRunningTime="2026-07-17 06:40:10.502611737 +0000 UTC m=+3537.691992020" Jul 17 06:40:10.998419 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:10.998377 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" podUID="3d79c01b-d32b-4e39-bc51-19cec06bb576" containerName="main" containerID="cri-o://c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e" gracePeriod=45 Jul 17 06:40:11.259761 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.259702 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:40:11.337764 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.337932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.337932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q99q2\" (UniqueName: \"kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.337932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337843 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.337932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337866 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.337932 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.337931 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir\") pod \"3d79c01b-d32b-4e39-bc51-19cec06bb576\" (UID: \"3d79c01b-d32b-4e39-bc51-19cec06bb576\") " Jul 17 06:40:11.338188 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.338016 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:40:11.338188 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.338107 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home" (OuterVolumeSpecName: "home") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:40:11.338337 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.338279 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.338337 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.338298 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.338337 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.338291 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:40:11.340086 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.340056 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm" (OuterVolumeSpecName: "dshm") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:40:11.340223 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.340105 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:40:11.340223 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.340130 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2" (OuterVolumeSpecName: "kube-api-access-q99q2") pod "3d79c01b-d32b-4e39-bc51-19cec06bb576" (UID: "3d79c01b-d32b-4e39-bc51-19cec06bb576"). InnerVolumeSpecName "kube-api-access-q99q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:40:11.439714 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.439675 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.439714 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.439709 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d79c01b-d32b-4e39-bc51-19cec06bb576-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.439714 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.439719 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99q2\" (UniqueName: \"kubernetes.io/projected/3d79c01b-d32b-4e39-bc51-19cec06bb576-kube-api-access-q99q2\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.439941 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.439727 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d79c01b-d32b-4e39-bc51-19cec06bb576-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:40:11.491527 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.491498 2574 generic.go:356] "Generic (PLEG): container finished" podID="3d79c01b-d32b-4e39-bc51-19cec06bb576" containerID="c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e" exitCode=0 Jul 17 06:40:11.491706 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.491549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" Jul 17 06:40:11.491706 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.491578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" event={"ID":"3d79c01b-d32b-4e39-bc51-19cec06bb576","Type":"ContainerDied","Data":"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e"} Jul 17 06:40:11.491706 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.491618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq" event={"ID":"3d79c01b-d32b-4e39-bc51-19cec06bb576","Type":"ContainerDied","Data":"c7be31b982a4e295f7741e3535c482a0eeba0b5d95ee8991bfb4c31c51f76a52"} Jul 17 06:40:11.491706 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.491638 2574 scope.go:117] "RemoveContainer" containerID="c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e" Jul 17 06:40:11.502083 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.502068 2574 scope.go:117] "RemoveContainer" containerID="c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e" Jul 17 06:40:11.502348 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:40:11.502331 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e\": container with ID starting with c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e not found: ID does not exist" containerID="c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e" Jul 17 06:40:11.502400 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.502357 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e"} err="failed to get container status \"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e\": rpc error: code = NotFound desc = could not find container \"c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e\": container with ID starting with c55d2776dc0df2a9d2f3a4057cf204b3fa3a555bc06f6423dc7cde07990bb10e not found: ID does not exist" Jul 17 06:40:11.514077 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.514020 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:40:11.520741 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:11.520717 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/sub-test-model-ff4257-kserve-784d7f44b7-vf6pq"] Jul 17 06:40:13.283046 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:13.283015 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d79c01b-d32b-4e39-bc51-19cec06bb576" path="/var/lib/kubelet/pods/3d79c01b-d32b-4e39-bc51-19cec06bb576/volumes" Jul 17 06:40:20.188327 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:20.188297 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:20.188765 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:20.188339 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:20.201036 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:20.201011 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:40:20.555385 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:40:20.555307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:41:13.814387 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:13.814288 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:41:13.847788 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:13.847759 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lgqkw_aafc48fa-52d1-44c9-9b9d-004664557555/ovn-acl-logging/0.log" Jul 17 06:41:25.472556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:25.472499 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:41:27.377111 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.377076 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046697922352"] Jul 17 06:41:27.377529 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.377451 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d79c01b-d32b-4e39-bc51-19cec06bb576" containerName="main" Jul 17 06:41:27.377529 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.377462 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d79c01b-d32b-4e39-bc51-19cec06bb576" containerName="main" Jul 17 06:41:27.377529 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.377522 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d79c01b-d32b-4e39-bc51-19cec06bb576" containerName="main" Jul 17 06:41:27.380434 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.380416 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:27.382958 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.382928 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kqqsp\"" Jul 17 06:41:27.389064 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.389039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697922352"] Jul 17 06:41:27.512768 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.512736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cph\" (UniqueName: \"kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph\") pod \"test-curl-7817-140046697922352\" (UID: \"c26a60bf-892e-4eaf-beb8-5f5451aa7f81\") " pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:27.613129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.613095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cph\" (UniqueName: \"kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph\") pod \"test-curl-7817-140046697922352\" (UID: \"c26a60bf-892e-4eaf-beb8-5f5451aa7f81\") " pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:27.622900 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.622874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cph\" (UniqueName: \"kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph\") pod \"test-curl-7817-140046697922352\" (UID: \"c26a60bf-892e-4eaf-beb8-5f5451aa7f81\") " pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:27.691113 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.691041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:27.827397 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:27.827364 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc26a60bf_892e_4eaf_beb8_5f5451aa7f81.slice/crio-8550a8c51f28e96332aa157ce55f672e944b6243411ad3f875a13961861b6c52 WatchSource:0}: Error finding container 8550a8c51f28e96332aa157ce55f672e944b6243411ad3f875a13961861b6c52: Status 404 returned error can't find the container with id 8550a8c51f28e96332aa157ce55f672e944b6243411ad3f875a13961861b6c52 Jul 17 06:41:27.827550 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.827527 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697922352"] Jul 17 06:41:27.828835 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.828818 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 06:41:27.913666 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:27.913639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697922352" event={"ID":"c26a60bf-892e-4eaf-beb8-5f5451aa7f81","Type":"ContainerStarted","Data":"8550a8c51f28e96332aa157ce55f672e944b6243411ad3f875a13961861b6c52"} Jul 17 06:41:28.920000 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:28.919967 2574 generic.go:356] "Generic (PLEG): container finished" podID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" containerID="d32e1d604e8ba376e917f79f79aec7e34400f83c71b3bd62e18be813467cc8fc" exitCode=0 Jul 17 06:41:28.920429 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:28.920062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697922352" event={"ID":"c26a60bf-892e-4eaf-beb8-5f5451aa7f81","Type":"ContainerDied","Data":"d32e1d604e8ba376e917f79f79aec7e34400f83c71b3bd62e18be813467cc8fc"} Jul 17 06:41:30.063018 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.062996 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:30.100656 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.100626 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046697922352_c26a60bf-892e-4eaf-beb8-5f5451aa7f81/test-curl-7817-140046697922352/0.log" Jul 17 06:41:30.123922 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.123891 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046697922352"] Jul 17 06:41:30.127811 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.127789 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046697922352"] Jul 17 06:41:30.235048 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.234963 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9cph\" (UniqueName: \"kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph\") pod \"c26a60bf-892e-4eaf-beb8-5f5451aa7f81\" (UID: \"c26a60bf-892e-4eaf-beb8-5f5451aa7f81\") " Jul 17 06:41:30.237590 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.237545 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph" (OuterVolumeSpecName: "kube-api-access-d9cph") pod "c26a60bf-892e-4eaf-beb8-5f5451aa7f81" (UID: "c26a60bf-892e-4eaf-beb8-5f5451aa7f81"). InnerVolumeSpecName "kube-api-access-d9cph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:30.246315 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.246290 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046697890352"] Jul 17 06:41:30.246643 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.246631 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" containerName="test-curl-7817-140046697922352" Jul 17 06:41:30.246694 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.246645 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" containerName="test-curl-7817-140046697922352" Jul 17 06:41:30.246739 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.246715 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" containerName="test-curl-7817-140046697922352" Jul 17 06:41:30.249888 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.249869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:30.256527 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.256501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697890352"] Jul 17 06:41:30.336344 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.336313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxzt\" (UniqueName: \"kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt\") pod \"test-curl-7817-140046697890352\" (UID: \"29a5aed8-4526-48c2-8a59-0aae85984b56\") " pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:30.336503 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.336422 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9cph\" (UniqueName: \"kubernetes.io/projected/c26a60bf-892e-4eaf-beb8-5f5451aa7f81-kube-api-access-d9cph\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:30.437344 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.437305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxzt\" (UniqueName: \"kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt\") pod \"test-curl-7817-140046697890352\" (UID: \"29a5aed8-4526-48c2-8a59-0aae85984b56\") " pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:30.445911 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.445885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxzt\" (UniqueName: \"kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt\") pod \"test-curl-7817-140046697890352\" (UID: \"29a5aed8-4526-48c2-8a59-0aae85984b56\") " pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:30.560751 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.560663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:30.694948 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.694922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697890352"] Jul 17 06:41:30.697581 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:30.697544 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a5aed8_4526_48c2_8a59_0aae85984b56.slice/crio-fbfb34af9932257d022fc38ec1a6cce03c522c437b5b9d832201bc4a0be8f501 WatchSource:0}: Error finding container fbfb34af9932257d022fc38ec1a6cce03c522c437b5b9d832201bc4a0be8f501: Status 404 returned error can't find the container with id fbfb34af9932257d022fc38ec1a6cce03c522c437b5b9d832201bc4a0be8f501 Jul 17 06:41:30.932328 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.932289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697890352" event={"ID":"29a5aed8-4526-48c2-8a59-0aae85984b56","Type":"ContainerStarted","Data":"fbfb34af9932257d022fc38ec1a6cce03c522c437b5b9d832201bc4a0be8f501"} Jul 17 06:41:30.933541 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.933525 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8550a8c51f28e96332aa157ce55f672e944b6243411ad3f875a13961861b6c52" Jul 17 06:41:30.933541 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:30.933533 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:30.943781 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:41:30.943754 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7817-140046697922352\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" pod="opendatahub/test-curl-7817-140046697922352" Jul 17 06:41:31.289038 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:31.288956 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26a60bf-892e-4eaf-beb8-5f5451aa7f81" path="/var/lib/kubelet/pods/c26a60bf-892e-4eaf-beb8-5f5451aa7f81/volumes" Jul 17 06:41:31.940174 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:31.940142 2574 generic.go:356] "Generic (PLEG): container finished" podID="29a5aed8-4526-48c2-8a59-0aae85984b56" containerID="9954f31a7301216c4163d773ef1ed2df9a863f4a70eec8c8ac5278eb5af8c9ba" exitCode=0 Jul 17 06:41:31.940385 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:31.940219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697890352" event={"ID":"29a5aed8-4526-48c2-8a59-0aae85984b56","Type":"ContainerDied","Data":"9954f31a7301216c4163d773ef1ed2df9a863f4a70eec8c8ac5278eb5af8c9ba"} Jul 17 06:41:33.077436 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.077410 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:33.098802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.098779 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046697890352_29a5aed8-4526-48c2-8a59-0aae85984b56/test-curl-7817-140046697890352/0.log" Jul 17 06:41:33.125147 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.125118 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046697890352"] Jul 17 06:41:33.129305 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.129283 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046697890352"] Jul 17 06:41:33.161301 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.161276 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrxzt\" (UniqueName: \"kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt\") pod \"29a5aed8-4526-48c2-8a59-0aae85984b56\" (UID: \"29a5aed8-4526-48c2-8a59-0aae85984b56\") " Jul 17 06:41:33.163456 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.163436 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt" (OuterVolumeSpecName: "kube-api-access-zrxzt") pod "29a5aed8-4526-48c2-8a59-0aae85984b56" (UID: "29a5aed8-4526-48c2-8a59-0aae85984b56"). InnerVolumeSpecName "kube-api-access-zrxzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:33.262202 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.262126 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxzt\" (UniqueName: \"kubernetes.io/projected/29a5aed8-4526-48c2-8a59-0aae85984b56-kube-api-access-zrxzt\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:33.282404 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.282376 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a5aed8-4526-48c2-8a59-0aae85984b56" path="/var/lib/kubelet/pods/29a5aed8-4526-48c2-8a59-0aae85984b56/volumes" Jul 17 06:41:33.282720 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.282706 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:33.283030 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.283019 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a5aed8-4526-48c2-8a59-0aae85984b56" containerName="test-curl-7817-140046697890352" Jul 17 06:41:33.283072 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.283031 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a5aed8-4526-48c2-8a59-0aae85984b56" containerName="test-curl-7817-140046697890352" Jul 17 06:41:33.283113 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.283094 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a5aed8-4526-48c2-8a59-0aae85984b56" containerName="test-curl-7817-140046697890352" Jul 17 06:41:33.286130 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.286109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:33.288798 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.288775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:33.464320 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.464285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswnc\" (UniqueName: \"kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc\") pod \"test-curl-7817-140046697856432\" (UID: \"49bba64e-a09c-4103-b242-e8514794a9c7\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:33.565625 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.565534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswnc\" (UniqueName: \"kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc\") pod \"test-curl-7817-140046697856432\" (UID: \"49bba64e-a09c-4103-b242-e8514794a9c7\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:33.573827 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.573801 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswnc\" (UniqueName: \"kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc\") pod \"test-curl-7817-140046697856432\" (UID: \"49bba64e-a09c-4103-b242-e8514794a9c7\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:33.597838 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.597812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:33.730664 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.730641 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:33.732038 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:33.732005 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49bba64e_a09c_4103_b242_e8514794a9c7.slice/crio-48f623b65d7ac5c81828e30077cd7a0e8a3c250b35cfc7a107dbdd55f76fa66f WatchSource:0}: Error finding container 48f623b65d7ac5c81828e30077cd7a0e8a3c250b35cfc7a107dbdd55f76fa66f: Status 404 returned error can't find the container with id 48f623b65d7ac5c81828e30077cd7a0e8a3c250b35cfc7a107dbdd55f76fa66f Jul 17 06:41:33.952224 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.952185 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697856432" event={"ID":"49bba64e-a09c-4103-b242-e8514794a9c7","Type":"ContainerStarted","Data":"48f623b65d7ac5c81828e30077cd7a0e8a3c250b35cfc7a107dbdd55f76fa66f"} Jul 17 06:41:33.953597 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.953579 2574 scope.go:117] "RemoveContainer" containerID="9954f31a7301216c4163d773ef1ed2df9a863f4a70eec8c8ac5278eb5af8c9ba" Jul 17 06:41:33.953719 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:33.953598 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697890352" Jul 17 06:41:34.960725 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:34.960687 2574 generic.go:356] "Generic (PLEG): container finished" podID="49bba64e-a09c-4103-b242-e8514794a9c7" containerID="36b7ab898d8456e6c28a5a1327c6dbc6c4d9ce964342ca29de89f1446a5601aa" exitCode=0 Jul 17 06:41:34.961118 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:34.960768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697856432" event={"ID":"49bba64e-a09c-4103-b242-e8514794a9c7","Type":"ContainerDied","Data":"36b7ab898d8456e6c28a5a1327c6dbc6c4d9ce964342ca29de89f1446a5601aa"} Jul 17 06:41:36.099985 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.099959 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.119397 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.119367 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046697856432_49bba64e-a09c-4103-b242-e8514794a9c7/test-curl-7817-140046697856432/0.log" Jul 17 06:41:36.140284 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.140235 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:36.143778 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.143742 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:36.189552 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.189527 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dswnc\" (UniqueName: \"kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc\") pod \"49bba64e-a09c-4103-b242-e8514794a9c7\" (UID: \"49bba64e-a09c-4103-b242-e8514794a9c7\") " Jul 17 06:41:36.191665 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.191642 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc" (OuterVolumeSpecName: "kube-api-access-dswnc") pod "49bba64e-a09c-4103-b242-e8514794a9c7" (UID: "49bba64e-a09c-4103-b242-e8514794a9c7"). InnerVolumeSpecName "kube-api-access-dswnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:36.261006 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.260932 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:36.261329 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.261315 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bba64e-a09c-4103-b242-e8514794a9c7" containerName="test-curl-7817-140046697856432" Jul 17 06:41:36.261382 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.261331 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bba64e-a09c-4103-b242-e8514794a9c7" containerName="test-curl-7817-140046697856432" Jul 17 06:41:36.261435 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.261404 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49bba64e-a09c-4103-b242-e8514794a9c7" containerName="test-curl-7817-140046697856432" Jul 17 06:41:36.264153 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.264138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.271216 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.271190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:36.290500 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.290477 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswnc\" (UniqueName: \"kubernetes.io/projected/49bba64e-a09c-4103-b242-e8514794a9c7-kube-api-access-dswnc\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:36.391479 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.391451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxp4\" (UniqueName: \"kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4\") pod \"test-curl-7817-140046697856432\" (UID: \"2e5e689a-4ce5-460d-b36e-33e0535f52a1\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.492555 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.492525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxp4\" (UniqueName: \"kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4\") pod \"test-curl-7817-140046697856432\" (UID: \"2e5e689a-4ce5-460d-b36e-33e0535f52a1\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.503802 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.503777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxp4\" (UniqueName: \"kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4\") pod \"test-curl-7817-140046697856432\" (UID: \"2e5e689a-4ce5-460d-b36e-33e0535f52a1\") " pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.575001 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.574916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.711716 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.711692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:36.714334 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:36.714306 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e5e689a_4ce5_460d_b36e_33e0535f52a1.slice/crio-61123b123d743f79e89fa79fc452e20dd2c8bfd53bc3977ca8232a0e18f2f30f WatchSource:0}: Error finding container 61123b123d743f79e89fa79fc452e20dd2c8bfd53bc3977ca8232a0e18f2f30f: Status 404 returned error can't find the container with id 61123b123d743f79e89fa79fc452e20dd2c8bfd53bc3977ca8232a0e18f2f30f Jul 17 06:41:36.972448 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.972415 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:36.972611 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.972427 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f623b65d7ac5c81828e30077cd7a0e8a3c250b35cfc7a107dbdd55f76fa66f" Jul 17 06:41:36.973598 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.973567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697856432" event={"ID":"2e5e689a-4ce5-460d-b36e-33e0535f52a1","Type":"ContainerStarted","Data":"61123b123d743f79e89fa79fc452e20dd2c8bfd53bc3977ca8232a0e18f2f30f"} Jul 17 06:41:36.984368 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:36.984340 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7817-140046697856432" oldPodUID="49bba64e-a09c-4103-b242-e8514794a9c7" podUID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" Jul 17 06:41:37.282704 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:37.282625 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49bba64e-a09c-4103-b242-e8514794a9c7" path="/var/lib/kubelet/pods/49bba64e-a09c-4103-b242-e8514794a9c7/volumes" Jul 17 06:41:37.980613 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:37.980580 2574 generic.go:356] "Generic (PLEG): container finished" podID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" containerID="0b70e0b096c3ceef386c6b5e6fcd24004a6bd03510a940089a17182b0aa5b362" exitCode=0 Jul 17 06:41:37.980792 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:37.980630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046697856432" event={"ID":"2e5e689a-4ce5-460d-b36e-33e0535f52a1","Type":"ContainerDied","Data":"0b70e0b096c3ceef386c6b5e6fcd24004a6bd03510a940089a17182b0aa5b362"} Jul 17 06:41:39.146303 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.146280 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:39.166015 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.165994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046697856432_2e5e689a-4ce5-460d-b36e-33e0535f52a1/test-curl-7817-140046697856432/0.log" Jul 17 06:41:39.189599 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.189569 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:39.193638 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.193616 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046697856432"] Jul 17 06:41:39.313970 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.313896 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:39.314287 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.314272 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" containerName="test-curl-7817-140046697856432" Jul 17 06:41:39.314343 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.314288 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" containerName="test-curl-7817-140046697856432" Jul 17 06:41:39.314400 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.314388 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" containerName="test-curl-7817-140046697856432" Jul 17 06:41:39.316981 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.316965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:39.318833 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.318808 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfxp4\" (UniqueName: \"kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4\") pod \"2e5e689a-4ce5-460d-b36e-33e0535f52a1\" (UID: \"2e5e689a-4ce5-460d-b36e-33e0535f52a1\") " Jul 17 06:41:39.321556 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.321534 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4" (OuterVolumeSpecName: "kube-api-access-qfxp4") pod "2e5e689a-4ce5-460d-b36e-33e0535f52a1" (UID: "2e5e689a-4ce5-460d-b36e-33e0535f52a1"). InnerVolumeSpecName "kube-api-access-qfxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:39.323205 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.323184 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:39.419854 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.419809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlqn\" (UniqueName: \"kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn\") pod \"test-curl-7817-140046695481824\" (UID: \"44d16a61-5276-4d05-bb75-99fcc4171ba8\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:39.420017 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.419939 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxp4\" (UniqueName: \"kubernetes.io/projected/2e5e689a-4ce5-460d-b36e-33e0535f52a1-kube-api-access-qfxp4\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:39.520388 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.520348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlqn\" (UniqueName: \"kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn\") pod \"test-curl-7817-140046695481824\" (UID: \"44d16a61-5276-4d05-bb75-99fcc4171ba8\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:39.529284 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.529234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlqn\" (UniqueName: \"kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn\") pod \"test-curl-7817-140046695481824\" (UID: \"44d16a61-5276-4d05-bb75-99fcc4171ba8\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:39.635301 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.635274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:39.768366 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.768336 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:39.770280 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:39.770221 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d16a61_5276_4d05_bb75_99fcc4171ba8.slice/crio-3ea0ad8b45785fde9e25db10a98b5cc1ee64801ad644c16d7efc086d9194d494 WatchSource:0}: Error finding container 3ea0ad8b45785fde9e25db10a98b5cc1ee64801ad644c16d7efc086d9194d494: Status 404 returned error can't find the container with id 3ea0ad8b45785fde9e25db10a98b5cc1ee64801ad644c16d7efc086d9194d494 Jul 17 06:41:39.992363 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.992280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046695481824" event={"ID":"44d16a61-5276-4d05-bb75-99fcc4171ba8","Type":"ContainerStarted","Data":"3ea0ad8b45785fde9e25db10a98b5cc1ee64801ad644c16d7efc086d9194d494"} Jul 17 06:41:39.993485 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.993466 2574 scope.go:117] "RemoveContainer" containerID="0b70e0b096c3ceef386c6b5e6fcd24004a6bd03510a940089a17182b0aa5b362" Jul 17 06:41:39.993612 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:39.993495 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046697856432" Jul 17 06:41:40.493128 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.493096 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerName="main" containerID="cri-o://7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe" gracePeriod=45 Jul 17 06:41:40.544847 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.544810 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Jul 17 06:41:40.748192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.748137 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:41:40.932102 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932062 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932139 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932222 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932240 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7stx\" (UniqueName: \"kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx\") pod \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\" (UID: \"b221d014-4676-4db7-beb7-8f5aa4a7e8a9\") " Jul 17 06:41:40.932555 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932432 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:41:40.932555 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:41:40.932555 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932470 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home" (OuterVolumeSpecName: "home") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:41:40.932711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932572 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:40.932711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932588 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:40.932711 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.932596 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:40.934532 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.934507 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 06:41:40.934532 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.934518 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm" (OuterVolumeSpecName: "dshm") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 06:41:40.934679 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:40.934603 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx" (OuterVolumeSpecName: "kube-api-access-q7stx") pod "b221d014-4676-4db7-beb7-8f5aa4a7e8a9" (UID: "b221d014-4676-4db7-beb7-8f5aa4a7e8a9"). InnerVolumeSpecName "kube-api-access-q7stx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:41.000298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.000201 2574 generic.go:356] "Generic (PLEG): container finished" podID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerID="7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe" exitCode=0 Jul 17 06:41:41.000298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.000265 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" Jul 17 06:41:41.000489 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.000295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" event={"ID":"b221d014-4676-4db7-beb7-8f5aa4a7e8a9","Type":"ContainerDied","Data":"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe"} Jul 17 06:41:41.000489 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.000344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p" event={"ID":"b221d014-4676-4db7-beb7-8f5aa4a7e8a9","Type":"ContainerDied","Data":"f1a303625c3800c18ce4c234ec4e5b4f4e32ede765c0c6e7f515c33a86f5a1d1"} Jul 17 06:41:41.000489 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.000369 2574 scope.go:117] "RemoveContainer" containerID="7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe" Jul 17 06:41:41.003016 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.002992 2574 generic.go:356] "Generic (PLEG): container finished" podID="44d16a61-5276-4d05-bb75-99fcc4171ba8" containerID="1566a5801039faaab96fd84fcf0a80cf4ca2e74ae74afb16a068474dfa5a8425" exitCode=0 Jul 17 06:41:41.003156 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.003016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046695481824" event={"ID":"44d16a61-5276-4d05-bb75-99fcc4171ba8","Type":"ContainerDied","Data":"1566a5801039faaab96fd84fcf0a80cf4ca2e74ae74afb16a068474dfa5a8425"} Jul 17 06:41:41.010544 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.010523 2574 scope.go:117] "RemoveContainer" containerID="7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe" Jul 17 06:41:41.010789 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:41:41.010772 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe\": container with ID starting with 7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe not found: ID does not exist" containerID="7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe" Jul 17 06:41:41.010854 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.010797 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe"} err="failed to get container status \"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe\": rpc error: code = NotFound desc = could not find container \"7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe\": container with ID starting with 7881d3a90b41f10178e6940c7cc27f288fca4d92cefe7b7ad6ceab39b3bdabfe not found: ID does not exist" Jul 17 06:41:41.033467 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.033440 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:41.033594 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.033489 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:41.033594 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.033503 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7stx\" (UniqueName: \"kubernetes.io/projected/b221d014-4676-4db7-beb7-8f5aa4a7e8a9-kube-api-access-q7stx\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:41.038735 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.038708 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:41:41.044045 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.044013 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ff4257/rate-test-model-ff4257-kserve-69df4459bc-bdw4p"] Jul 17 06:41:41.283596 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.283512 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5e689a-4ce5-460d-b36e-33e0535f52a1" path="/var/lib/kubelet/pods/2e5e689a-4ce5-460d-b36e-33e0535f52a1/volumes" Jul 17 06:41:41.283895 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:41.283883 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" path="/var/lib/kubelet/pods/b221d014-4676-4db7-beb7-8f5aa4a7e8a9/volumes" Jul 17 06:41:42.143212 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.143192 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.163306 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.163284 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046695481824_44d16a61-5276-4d05-bb75-99fcc4171ba8/test-curl-7817-140046695481824/0.log" Jul 17 06:41:42.187800 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.187766 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:42.192232 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.192211 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:42.243367 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.243342 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrlqn\" (UniqueName: \"kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn\") pod \"44d16a61-5276-4d05-bb75-99fcc4171ba8\" (UID: \"44d16a61-5276-4d05-bb75-99fcc4171ba8\") " Jul 17 06:41:42.245490 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.245463 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn" (OuterVolumeSpecName: "kube-api-access-rrlqn") pod "44d16a61-5276-4d05-bb75-99fcc4171ba8" (UID: "44d16a61-5276-4d05-bb75-99fcc4171ba8"). InnerVolumeSpecName "kube-api-access-rrlqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:42.317284 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317204 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:42.317746 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317726 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d16a61-5276-4d05-bb75-99fcc4171ba8" containerName="test-curl-7817-140046695481824" Jul 17 06:41:42.317786 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317752 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d16a61-5276-4d05-bb75-99fcc4171ba8" containerName="test-curl-7817-140046695481824" Jul 17 06:41:42.317786 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317782 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerName="main" Jul 17 06:41:42.317867 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317790 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerName="main" Jul 17 06:41:42.317919 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317880 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d16a61-5276-4d05-bb75-99fcc4171ba8" containerName="test-curl-7817-140046695481824" Jul 17 06:41:42.317919 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.317891 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b221d014-4676-4db7-beb7-8f5aa4a7e8a9" containerName="main" Jul 17 06:41:42.320839 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.320824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.335586 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.335563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:42.344750 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.344720 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrlqn\" (UniqueName: \"kubernetes.io/projected/44d16a61-5276-4d05-bb75-99fcc4171ba8-kube-api-access-rrlqn\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:42.445689 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.445659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99j7p\" (UniqueName: \"kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p\") pod \"test-curl-7817-140046695481824\" (UID: \"dd258919-d328-4e6c-85ab-c2d15752fbd0\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.546131 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.546100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99j7p\" (UniqueName: \"kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p\") pod \"test-curl-7817-140046695481824\" (UID: \"dd258919-d328-4e6c-85ab-c2d15752fbd0\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.554531 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.554503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99j7p\" (UniqueName: \"kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p\") pod \"test-curl-7817-140046695481824\" (UID: \"dd258919-d328-4e6c-85ab-c2d15752fbd0\") " pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.629178 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.629155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:42.767126 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:42.767100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:42.769080 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:41:42.769053 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd258919_d328_4e6c_85ab_c2d15752fbd0.slice/crio-06a78f9bd7a2a0094fd83a97a91c7b769cf9b2532015ae0c42f00da28fbdeefe WatchSource:0}: Error finding container 06a78f9bd7a2a0094fd83a97a91c7b769cf9b2532015ae0c42f00da28fbdeefe: Status 404 returned error can't find the container with id 06a78f9bd7a2a0094fd83a97a91c7b769cf9b2532015ae0c42f00da28fbdeefe Jul 17 06:41:43.016579 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:43.016484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046695481824" event={"ID":"dd258919-d328-4e6c-85ab-c2d15752fbd0","Type":"ContainerStarted","Data":"06a78f9bd7a2a0094fd83a97a91c7b769cf9b2532015ae0c42f00da28fbdeefe"} Jul 17 06:41:43.017645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:43.017621 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:43.017801 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:43.017666 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea0ad8b45785fde9e25db10a98b5cc1ee64801ad644c16d7efc086d9194d494" Jul 17 06:41:43.029200 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:43.029177 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7817-140046695481824" oldPodUID="44d16a61-5276-4d05-bb75-99fcc4171ba8" podUID="dd258919-d328-4e6c-85ab-c2d15752fbd0" Jul 17 06:41:43.282428 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:43.282349 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d16a61-5276-4d05-bb75-99fcc4171ba8" path="/var/lib/kubelet/pods/44d16a61-5276-4d05-bb75-99fcc4171ba8/volumes" Jul 17 06:41:44.027990 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:44.027958 2574 generic.go:356] "Generic (PLEG): container finished" podID="dd258919-d328-4e6c-85ab-c2d15752fbd0" containerID="fa0cbedbbdde4eee1d2a92f8750183ad26d23eeb02ed7d33762f53049e038201" exitCode=0 Jul 17 06:41:44.028192 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:44.028010 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7817-140046695481824" event={"ID":"dd258919-d328-4e6c-85ab-c2d15752fbd0","Type":"ContainerDied","Data":"fa0cbedbbdde4eee1d2a92f8750183ad26d23eeb02ed7d33762f53049e038201"} Jul 17 06:41:45.170603 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.170581 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:45.196334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.196306 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7817-140046695481824_dd258919-d328-4e6c-85ab-c2d15752fbd0/test-curl-7817-140046695481824/0.log" Jul 17 06:41:45.225274 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.225216 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:45.228810 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.228784 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7817-140046695481824"] Jul 17 06:41:45.268838 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.268807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99j7p\" (UniqueName: \"kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p\") pod \"dd258919-d328-4e6c-85ab-c2d15752fbd0\" (UID: \"dd258919-d328-4e6c-85ab-c2d15752fbd0\") " Jul 17 06:41:45.271226 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.271193 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p" (OuterVolumeSpecName: "kube-api-access-99j7p") pod "dd258919-d328-4e6c-85ab-c2d15752fbd0" (UID: "dd258919-d328-4e6c-85ab-c2d15752fbd0"). InnerVolumeSpecName "kube-api-access-99j7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 06:41:45.282560 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.282535 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd258919-d328-4e6c-85ab-c2d15752fbd0" path="/var/lib/kubelet/pods/dd258919-d328-4e6c-85ab-c2d15752fbd0/volumes" Jul 17 06:41:45.369934 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:45.369906 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99j7p\" (UniqueName: \"kubernetes.io/projected/dd258919-d328-4e6c-85ab-c2d15752fbd0-kube-api-access-99j7p\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:41:46.040676 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:46.040638 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7817-140046695481824" Jul 17 06:41:46.040676 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:41:46.040677 2574 scope.go:117] "RemoveContainer" containerID="fa0cbedbbdde4eee1d2a92f8750183ad26d23eeb02ed7d33762f53049e038201" Jul 17 06:42:06.883094 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.883062 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:06.883499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.883455 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd258919-d328-4e6c-85ab-c2d15752fbd0" containerName="test-curl-7817-140046695481824" Jul 17 06:42:06.883499 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.883468 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd258919-d328-4e6c-85ab-c2d15752fbd0" containerName="test-curl-7817-140046695481824" Jul 17 06:42:06.883578 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.883538 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd258919-d328-4e6c-85ab-c2d15752fbd0" containerName="test-curl-7817-140046695481824" Jul 17 06:42:06.888378 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.888362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:06.893750 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.893726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mtfhw\"" Jul 17 06:42:06.898298 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.898272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:06.960677 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:06.960647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6\" (UID: \"e2126d0e-169b-4630-bddb-c9bd2b786e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:07.061393 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.061359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6\" (UID: \"e2126d0e-169b-4630-bddb-c9bd2b786e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:07.061935 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.061917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6\" (UID: \"e2126d0e-169b-4630-bddb-c9bd2b786e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:07.200334 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.200257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:07.349603 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.349570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:07.351009 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:42:07.350983 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2126d0e_169b_4630_bddb_c9bd2b786e30.slice/crio-194f78463c618be53508a52af5a7156224cdc9ec949dfd7ee0fde81c04d57456 WatchSource:0}: Error finding container 194f78463c618be53508a52af5a7156224cdc9ec949dfd7ee0fde81c04d57456: Status 404 returned error can't find the container with id 194f78463c618be53508a52af5a7156224cdc9ec949dfd7ee0fde81c04d57456 Jul 17 06:42:07.796417 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.796384 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k"] Jul 17 06:42:07.799882 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.799867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:07.808438 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.808416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k"] Jul 17 06:42:07.868322 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.868295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k\" (UID: \"a613e5c7-8e2f-4b04-a2f1-9b52230f7762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:07.968832 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.968749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k\" (UID: \"a613e5c7-8e2f-4b04-a2f1-9b52230f7762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:07.969336 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:07.969319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k\" (UID: \"a613e5c7-8e2f-4b04-a2f1-9b52230f7762\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:08.112036 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.111997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:08.163781 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.163750 2574 generic.go:356] "Generic (PLEG): container finished" podID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerID="fc9b2cafa0c45a199c696ec2eacba8bb9b1d60d830002303be968cbbf08ddb75" exitCode=6 Jul 17 06:42:08.163958 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.163938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" event={"ID":"e2126d0e-169b-4630-bddb-c9bd2b786e30","Type":"ContainerDied","Data":"fc9b2cafa0c45a199c696ec2eacba8bb9b1d60d830002303be968cbbf08ddb75"} Jul 17 06:42:08.164100 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.163995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" event={"ID":"e2126d0e-169b-4630-bddb-c9bd2b786e30","Type":"ContainerStarted","Data":"194f78463c618be53508a52af5a7156224cdc9ec949dfd7ee0fde81c04d57456"} Jul 17 06:42:08.164197 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.164129 2574 scope.go:117] "RemoveContainer" containerID="fc9b2cafa0c45a199c696ec2eacba8bb9b1d60d830002303be968cbbf08ddb75" Jul 17 06:42:08.255230 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:08.255207 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k"] Jul 17 06:42:08.257129 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:42:08.257098 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda613e5c7_8e2f_4b04_a2f1_9b52230f7762.slice/crio-d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d WatchSource:0}: Error finding container d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d: Status 404 returned error can't find the container with id d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d Jul 17 06:42:09.171719 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.171643 2574 generic.go:356] "Generic (PLEG): container finished" podID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerID="11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0" exitCode=6 Jul 17 06:42:09.172126 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.171723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" event={"ID":"e2126d0e-169b-4630-bddb-c9bd2b786e30","Type":"ContainerDied","Data":"11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0"} Jul 17 06:42:09.172126 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.171785 2574 scope.go:117] "RemoveContainer" containerID="fc9b2cafa0c45a199c696ec2eacba8bb9b1d60d830002303be968cbbf08ddb75" Jul 17 06:42:09.172126 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.171973 2574 scope.go:117] "RemoveContainer" containerID="11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0" Jul 17 06:42:09.172288 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:42:09.172243 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6_odh-ai-gateway-infra(e2126d0e-169b-4630-bddb-c9bd2b786e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" Jul 17 06:42:09.173377 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.173357 2574 generic.go:356] "Generic (PLEG): container finished" podID="a613e5c7-8e2f-4b04-a2f1-9b52230f7762" containerID="660de71250a13e0c49cfdaaa808bc301ba6ba790f841d6015c28d49edd9945ff" exitCode=0 Jul 17 06:42:09.173493 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.173403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" event={"ID":"a613e5c7-8e2f-4b04-a2f1-9b52230f7762","Type":"ContainerDied","Data":"660de71250a13e0c49cfdaaa808bc301ba6ba790f841d6015c28d49edd9945ff"} Jul 17 06:42:09.173493 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:09.173451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" event={"ID":"a613e5c7-8e2f-4b04-a2f1-9b52230f7762","Type":"ContainerStarted","Data":"d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d"} Jul 17 06:42:10.181117 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:10.181090 2574 scope.go:117] "RemoveContainer" containerID="11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0" Jul 17 06:42:10.181546 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:42:10.181322 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6_odh-ai-gateway-infra(e2126d0e-169b-4630-bddb-c9bd2b786e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" Jul 17 06:42:10.320576 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:10.320549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:10.390202 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:10.390178 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca\") pod \"a613e5c7-8e2f-4b04-a2f1-9b52230f7762\" (UID: \"a613e5c7-8e2f-4b04-a2f1-9b52230f7762\") " Jul 17 06:42:10.390525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:10.390506 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a613e5c7-8e2f-4b04-a2f1-9b52230f7762" (UID: "a613e5c7-8e2f-4b04-a2f1-9b52230f7762"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:10.491141 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:10.491066 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a613e5c7-8e2f-4b04-a2f1-9b52230f7762-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:42:11.187910 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:11.187879 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" Jul 17 06:42:11.187910 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:11.187895 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k" event={"ID":"a613e5c7-8e2f-4b04-a2f1-9b52230f7762","Type":"ContainerDied","Data":"d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d"} Jul 17 06:42:11.188405 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:11.187977 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1403f8b9f2371150419b45aadb471599ca8a770f11adf6d41bf2161fa84117d" Jul 17 06:42:21.277133 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:21.277098 2574 scope.go:117] "RemoveContainer" containerID="11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0" Jul 17 06:42:22.250885 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:22.250800 2574 generic.go:356] "Generic (PLEG): container finished" podID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerID="805963076d098d9342f24d652eae76d9b7f4494c32080b81b40edda0697e5d76" exitCode=6 Jul 17 06:42:22.251051 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:22.250874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" event={"ID":"e2126d0e-169b-4630-bddb-c9bd2b786e30","Type":"ContainerDied","Data":"805963076d098d9342f24d652eae76d9b7f4494c32080b81b40edda0697e5d76"} Jul 17 06:42:22.251051 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:22.250947 2574 scope.go:117] "RemoveContainer" containerID="11757e8a31965912a33bde040c21b1b2d8d5bbf8f293610a12048bebd37b2cb0" Jul 17 06:42:22.251313 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:22.251297 2574 scope.go:117] "RemoveContainer" containerID="805963076d098d9342f24d652eae76d9b7f4494c32080b81b40edda0697e5d76" Jul 17 06:42:22.251550 ip-10-0-136-88 kubenswrapper[2574]: E0717 06:42:22.251523 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6_odh-ai-gateway-infra(e2126d0e-169b-4630-bddb-c9bd2b786e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" Jul 17 06:42:23.286772 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:23.286740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:23.422730 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:23.422710 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:23.505024 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:23.504995 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca\") pod \"e2126d0e-169b-4630-bddb-c9bd2b786e30\" (UID: \"e2126d0e-169b-4630-bddb-c9bd2b786e30\") " Jul 17 06:42:23.505362 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:23.505342 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2126d0e-169b-4630-bddb-c9bd2b786e30" (UID: "e2126d0e-169b-4630-bddb-c9bd2b786e30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 06:42:23.606606 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:23.606570 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2126d0e-169b-4630-bddb-c9bd2b786e30-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\"" Jul 17 06:42:24.264104 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:24.264069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" event={"ID":"e2126d0e-169b-4630-bddb-c9bd2b786e30","Type":"ContainerDied","Data":"194f78463c618be53508a52af5a7156224cdc9ec949dfd7ee0fde81c04d57456"} Jul 17 06:42:24.264104 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:24.264095 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6" Jul 17 06:42:24.264393 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:24.264133 2574 scope.go:117] "RemoveContainer" containerID="805963076d098d9342f24d652eae76d9b7f4494c32080b81b40edda0697e5d76" Jul 17 06:42:24.291664 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:24.291641 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:24.299491 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:24.299469 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c567a84-lcsw6"] Jul 17 06:42:25.281977 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:25.281929 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" path="/var/lib/kubelet/pods/e2126d0e-169b-4630-bddb-c9bd2b786e30/volumes" Jul 17 06:42:31.155117 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:31.155087 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5774d84fdb-qdzd9_456064d9-05e9-433a-8055-6e917444af36/authorino/0.log" Jul 17 06:42:37.385990 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:37.385908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-mzhrl_80784fe4-77e9-4f12-92fe-06bbfbd6f1f2/copy-manifests/0.log" Jul 17 06:42:37.393902 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:37.393873 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-mzhrl_80784fe4-77e9-4f12-92fe-06bbfbd6f1f2/manager/0.log" Jul 17 06:42:38.485917 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:38.485886 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-f90cce-vqjxr_2c7e0537-1caa-4da1-96ee-18dfb3c12b89/revoke-keys/0.log" Jul 17 06:42:39.418149 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:39.418122 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-346c23-h8v8k_a613e5c7-8e2f-4b04-a2f1-9b52230f7762/revoke-keys/0.log" Jul 17 06:42:39.541543 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:39.541519 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-864zj_86a8f91b-b0be-42e8-b473-9dd1f30750d7/postgres/0.log" Jul 17 06:42:40.329305 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.329280 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/pull/0.log" Jul 17 06:42:40.346758 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.346738 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/extract/0.log" Jul 17 06:42:40.356642 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.356624 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/util/0.log" Jul 17 06:42:40.469312 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.469282 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/util/0.log" Jul 17 06:42:40.476601 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.476570 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/pull/0.log" Jul 17 06:42:40.482945 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.482927 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/extract/0.log" Jul 17 06:42:40.596049 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.595954 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/util/0.log" Jul 17 06:42:40.604771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.604748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/pull/0.log" Jul 17 06:42:40.612001 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.611984 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/extract/0.log" Jul 17 06:42:40.725339 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.725316 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/util/0.log" Jul 17 06:42:40.733936 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.733915 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/pull/0.log" Jul 17 06:42:40.742596 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.742575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/extract/0.log" Jul 17 06:42:40.861012 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:40.860930 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5774d84fdb-qdzd9_456064d9-05e9-433a-8055-6e917444af36/authorino/0.log" Jul 17 06:42:42.427966 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:42.427937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d55f4645-p5w2p_bc1dbc69-7342-4d58-ae94-4d84acdf7350/kube-auth-proxy/0.log" Jul 17 06:42:42.679136 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:42.679051 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bkl69_a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af/payload-pre-processing/0.log" Jul 17 06:42:42.797553 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:42.797520 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tprvq_b39145f8-c13e-4957-83d2-3acf8555813a/payload-processing/0.log" Jul 17 06:42:42.907337 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:42.907306 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687684fd5b-qjmnh_bef8682f-58cd-475a-a989-97a2a548fce0/router/0.log" Jul 17 06:42:43.373560 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:43.373524 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vvfnh_ebd9065b-f376-47c9-a27d-0efed5c1b427/main/0.log" Jul 17 06:42:43.858397 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:43.858360 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5vv9l_67329077-a9e8-433b-8f93-4b03cde5e357/main/0.log" Jul 17 06:42:51.946825 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:51.946799 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g7rcl_4f55d9ac-02da-4762-8b7d-c1b25fb93f74/global-pull-secret-syncer/0.log" Jul 17 06:42:52.193495 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:52.193465 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lswjg_54f7b06a-0ab5-4306-8c29-c0e86a4762cf/konnectivity-agent/0.log" Jul 17 06:42:52.235525 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:52.235451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-88.ec2.internal_3544f86c6a812cb27ca33d092eba6944/haproxy/0.log" Jul 17 06:42:56.452358 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.452317 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/extract/0.log" Jul 17 06:42:56.475693 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.475663 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/util/0.log" Jul 17 06:42:56.502399 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.502372 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75926cn5_5c2f9e11-811e-41c7-b3dd-b85dc31f0425/pull/0.log" Jul 17 06:42:56.534704 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.534684 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/extract/0.log" Jul 17 06:42:56.561266 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.561232 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/util/0.log" Jul 17 06:42:56.584645 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.584623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jsqxf_db628d4d-7904-4fdc-91e5-2fec37975821/pull/0.log" Jul 17 06:42:56.613264 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.613234 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/extract/0.log" Jul 17 06:42:56.645007 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.644989 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/util/0.log" Jul 17 06:42:56.667890 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.667868 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73n8rzk_52050454-892f-4b64-b86e-416d3efdf46e/pull/0.log" Jul 17 06:42:56.700015 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.699992 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/extract/0.log" Jul 17 06:42:56.749545 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.749484 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/util/0.log" Jul 17 06:42:56.771678 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:56.771652 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tstjl_e4a58965-19eb-4337-9b87-f4c81ecc92b4/pull/0.log" Jul 17 06:42:57.120108 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:57.120076 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5774d84fdb-qdzd9_456064d9-05e9-433a-8055-6e917444af36/authorino/0.log" Jul 17 06:42:59.362064 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.362040 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-bhndn_66ede3fa-03c0-4bfc-b099-865b673591ec/cluster-monitoring-operator/0.log" Jul 17 06:42:59.400983 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.400955 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xzqcd_2b603103-a2ad-440f-9f20-704f47915c0d/kube-state-metrics/0.log" Jul 17 06:42:59.423087 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.423063 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xzqcd_2b603103-a2ad-440f-9f20-704f47915c0d/kube-rbac-proxy-main/0.log" Jul 17 06:42:59.445411 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.445383 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xzqcd_2b603103-a2ad-440f-9f20-704f47915c0d/kube-rbac-proxy-self/0.log" Jul 17 06:42:59.616065 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.616042 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzxfk_da8479c5-33af-4dfb-856d-6605cf464a8a/node-exporter/0.log" Jul 17 06:42:59.638884 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.638865 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzxfk_da8479c5-33af-4dfb-856d-6605cf464a8a/kube-rbac-proxy/0.log" Jul 17 06:42:59.665669 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:42:59.665655 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fzxfk_da8479c5-33af-4dfb-856d-6605cf464a8a/init-textfile/0.log" Jul 17 06:43:00.063563 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:00.063474 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-m4qm2_d9712124-38f7-47d7-b035-9894209bac77/prometheus-operator/0.log" Jul 17 06:43:00.085332 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:00.085298 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-m4qm2_d9712124-38f7-47d7-b035-9894209bac77/kube-rbac-proxy/0.log" Jul 17 06:43:00.147884 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:00.147859 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bbd7d56d-jwqgp_9cb4570c-4dfd-4f95-9f6f-12e9fae27d79/telemeter-client/0.log" Jul 17 06:43:00.171072 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:00.171050 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bbd7d56d-jwqgp_9cb4570c-4dfd-4f95-9f6f-12e9fae27d79/reload/0.log" Jul 17 06:43:00.194742 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:00.194723 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bbd7d56d-jwqgp_9cb4570c-4dfd-4f95-9f6f-12e9fae27d79/kube-rbac-proxy/0.log" Jul 17 06:43:02.452605 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452574 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5"] Jul 17 06:43:02.452973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452951 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.452973 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452962 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452979 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a613e5c7-8e2f-4b04-a2f1-9b52230f7762" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452984 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a613e5c7-8e2f-4b04-a2f1-9b52230f7762" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452991 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.452999 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.453004 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453054 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.453009 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453276 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.453069 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.453276 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.453077 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a613e5c7-8e2f-4b04-a2f1-9b52230f7762" containerName="revoke-keys" Jul 17 06:43:02.453276 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.453085 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2126d0e-169b-4630-bddb-c9bd2b786e30" containerName="revoke-keys" Jul 17 06:43:02.456206 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.456184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.458818 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.458796 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbz7r\"/\"kube-root-ca.crt\"" Jul 17 06:43:02.459137 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.459114 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbz7r\"/\"openshift-service-ca.crt\"" Jul 17 06:43:02.459887 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.459868 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbz7r\"/\"default-dockercfg-rdkgz\"" Jul 17 06:43:02.473771 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.473748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5"] Jul 17 06:43:02.535587 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.535556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5kjh\" (UniqueName: \"kubernetes.io/projected/28dea0aa-8406-475f-9a10-a1b2be727527-kube-api-access-m5kjh\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.535757 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.535594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-sys\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.535757 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.535702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-podres\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.535757 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.535747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-proc\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.535870 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.535771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-lib-modules\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.636822 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.636788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-podres\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.636986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.636835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-proc\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.636986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.636899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-proc\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.636986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.636932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-lib-modules\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.636986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.636957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-podres\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.637129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.637008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5kjh\" (UniqueName: \"kubernetes.io/projected/28dea0aa-8406-475f-9a10-a1b2be727527-kube-api-access-m5kjh\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.637129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.637028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-sys\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.637129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.637052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-lib-modules\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.637129 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.637103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28dea0aa-8406-475f-9a10-a1b2be727527-sys\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.646799 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.646774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5kjh\" (UniqueName: \"kubernetes.io/projected/28dea0aa-8406-475f-9a10-a1b2be727527-kube-api-access-m5kjh\") pod \"perf-node-gather-daemonset-l67g5\" (UID: \"28dea0aa-8406-475f-9a10-a1b2be727527\") " pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.767307 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.767206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:02.898280 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.898241 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59df6c8b55-st5v5_45b32a69-423a-41ca-bbb9-773e2094ce81/console/0.log" Jul 17 06:43:02.934765 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.934734 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5"] Jul 17 06:43:02.936831 ip-10-0-136-88 kubenswrapper[2574]: W0717 06:43:02.936799 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28dea0aa_8406_475f_9a10_a1b2be727527.slice/crio-1f851033e8afd683da11f5057cc64e09984da87a7dd8df1ca68ae2ac2754397a WatchSource:0}: Error finding container 1f851033e8afd683da11f5057cc64e09984da87a7dd8df1ca68ae2ac2754397a: Status 404 returned error can't find the container with id 1f851033e8afd683da11f5057cc64e09984da87a7dd8df1ca68ae2ac2754397a Jul 17 06:43:02.977264 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:02.977214 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-wmplw_b785f4f1-6406-434e-8376-e74896762cd5/download-server/0.log" Jul 17 06:43:03.477286 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:03.477190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" event={"ID":"28dea0aa-8406-475f-9a10-a1b2be727527","Type":"ContainerStarted","Data":"21fbcf29147f57dd4a687043d000f39333d7aac515247f13526c46b66cf9151c"} Jul 17 06:43:03.477643 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:03.477361 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:03.477643 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:03.477403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" event={"ID":"28dea0aa-8406-475f-9a10-a1b2be727527","Type":"ContainerStarted","Data":"1f851033e8afd683da11f5057cc64e09984da87a7dd8df1ca68ae2ac2754397a"} Jul 17 06:43:03.509016 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:03.508966 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" podStartSLOduration=1.5089492089999998 podStartE2EDuration="1.508949209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 06:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 06:43:03.50499581 +0000 UTC m=+3710.694376092" watchObservedRunningTime="2026-07-17 06:43:03.508949209 +0000 UTC m=+3710.698329515" Jul 17 06:43:03.702353 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:03.702321 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-gp55n_b2c9e7f3-a354-47d7-9c89-9879813f0705/volume-data-source-validator/0.log" Jul 17 06:43:04.908219 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:04.908187 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2l9p_aa8fcf25-18bd-42b7-bc0b-bbd34870577a/dns/0.log" Jul 17 06:43:04.932139 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:04.932118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p2l9p_aa8fcf25-18bd-42b7-bc0b-bbd34870577a/kube-rbac-proxy/0.log" Jul 17 06:43:04.960294 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:04.960270 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dnm5g_cf333d5d-2436-4afe-910c-e79c9cd7fa72/dns-node-resolver/0.log" Jul 17 06:43:05.639591 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:05.639561 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pft2v_7dbdb628-ab21-484c-9b77-17e81ddd6698/node-ca/0.log" Jul 17 06:43:06.979966 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:06.979936 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d55f4645-p5w2p_bc1dbc69-7342-4d58-ae94-4d84acdf7350/kube-auth-proxy/0.log" Jul 17 06:43:07.129717 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:07.129687 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bkl69_a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af/payload-pre-processing/0.log" Jul 17 06:43:07.170068 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:07.170041 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tprvq_b39145f8-c13e-4957-83d2-3acf8555813a/payload-processing/0.log" Jul 17 06:43:07.204237 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:07.204211 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687684fd5b-qjmnh_bef8682f-58cd-475a-a989-97a2a548fce0/router/0.log" Jul 17 06:43:08.151456 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:08.151432 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tv449_d91c61b2-0c22-4fca-873b-f947862774c5/serve-healthcheck-canary/0.log" Jul 17 06:43:08.783718 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:08.783689 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-m5wkv_fcadf004-328f-4c25-bd51-34074c28ea9e/insights-operator/0.log" Jul 17 06:43:08.784403 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:08.784389 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-m5wkv_fcadf004-328f-4c25-bd51-34074c28ea9e/insights-operator/1.log" Jul 17 06:43:08.961988 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:08.961961 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv9vc_63a926d0-adb4-453a-b7c1-568f38da49f2/kube-rbac-proxy/0.log" Jul 17 06:43:09.002130 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:09.002109 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv9vc_63a926d0-adb4-453a-b7c1-568f38da49f2/exporter/0.log" Jul 17 06:43:09.055230 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:09.055161 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cv9vc_63a926d0-adb4-453a-b7c1-568f38da49f2/extractor/0.log" Jul 17 06:43:09.493676 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:09.493652 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbz7r/perf-node-gather-daemonset-l67g5" Jul 17 06:43:12.694712 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:12.694685 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d55f4645-p5w2p_bc1dbc69-7342-4d58-ae94-4d84acdf7350/kube-auth-proxy/0.log" Jul 17 06:43:12.875459 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:12.875434 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bkl69_a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af/payload-pre-processing/0.log" Jul 17 06:43:12.953958 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:12.953926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tprvq_b39145f8-c13e-4957-83d2-3acf8555813a/payload-processing/0.log" Jul 17 06:43:12.974300 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:12.974273 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-mzhrl_80784fe4-77e9-4f12-92fe-06bbfbd6f1f2/manager/0.log" Jul 17 06:43:12.983547 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:12.983525 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687684fd5b-qjmnh_bef8682f-58cd-475a-a989-97a2a548fce0/router/0.log" Jul 17 06:43:13.046173 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:13.046151 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554df894d5-mzhrl_80784fe4-77e9-4f12-92fe-06bbfbd6f1f2/copy-manifests/0.log" Jul 17 06:43:14.504576 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.504547 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79d55f4645-p5w2p_bc1dbc69-7342-4d58-ae94-4d84acdf7350/kube-auth-proxy/0.log" Jul 17 06:43:14.597986 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.597960 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-bkl69_a44f66c3-f8c5-4fe2-b5ce-7b0af4a775af/payload-pre-processing/0.log" Jul 17 06:43:14.629848 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.629826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tprvq_b39145f8-c13e-4957-83d2-3acf8555813a/payload-processing/0.log" Jul 17 06:43:14.658997 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.658977 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d9bdd98d9-lmdvg_81aa3bc8-ab3e-4b17-b958-bf8e174bc01d/manager/0.log" Jul 17 06:43:14.679371 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.679356 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-687684fd5b-qjmnh_bef8682f-58cd-475a-a989-97a2a548fce0/router/0.log" Jul 17 06:43:14.780620 ip-10-0-136-88 kubenswrapper[2574]: I0717 06:43:14.780593 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-d6xdt_3123a727-91f7-48de-9e24-fd8fd1166153/openshift-lws-operator/0.log"