Jul 06 22:24:02.732964 ip-10-0-139-147 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:24:04.250489 ip-10-0-139-147 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:24:04.250489 ip-10-0-139-147 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:24:04.250489 ip-10-0-139-147 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:24:04.250489 ip-10-0-139-147 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:24:04.250489 ip-10-0-139-147 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:24:04.252846 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.252760 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259072 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259093 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259097 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259100 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259104 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.259098 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259107 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259110 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259113 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259116 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259119 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259121 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259124 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259127 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259129 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259132 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259134 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259137 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259139 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259142 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259145 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259149 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259153 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259160 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259163 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.259325 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259166 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259168 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259171 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259174 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259177 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259179 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259181 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259185 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259188 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259191 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259193 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259196 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259198 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259201 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259203 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259207 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259209 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259212 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259215 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259217 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.259763 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259220 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259223 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259225 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259227 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259230 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259232 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259237 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259239 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259242 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259244 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259247 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259250 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259252 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259255 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259257 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259260 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259262 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259265 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259267 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259270 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.260263 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259272 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259275 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259277 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259280 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259282 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259285 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259287 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259291 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259294 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259296 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259299 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259302 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259305 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259308 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259310 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259313 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259315 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259318 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259321 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.260740 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259324 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259326 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259330 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259334 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259337 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259340 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259343 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259346 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259349 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259352 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259355 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259357 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259360 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259362 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259365 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.259368 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261024 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261033 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261036 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.261205 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261040 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261043 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261046 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261048 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261052 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261055 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261058 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261061 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261063 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261066 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261069 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261071 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261074 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261077 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261080 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261083 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261086 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261088 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261091 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261094 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.261700 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261096 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261099 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261102 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261104 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261107 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261110 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261112 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261115 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261117 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261120 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261122 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261127 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261130 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261134 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261137 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261140 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261142 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261146 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261149 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261151 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.262206 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261154 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261156 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261159 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261163 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261166 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261168 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261171 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261174 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261177 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261179 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261182 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261184 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261187 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261189 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261192 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261194 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261197 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261199 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261202 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261204 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.262725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261206 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261209 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261212 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261214 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261217 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261220 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261222 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261225 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261228 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261230 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261234 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261238 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261241 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261244 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261247 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261250 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261253 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261256 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261259 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.263232 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261262 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261265 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261268 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261271 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261274 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261277 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261279 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261282 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261284 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261287 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261289 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261292 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261295 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261299 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261302 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261305 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261307 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261720 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261726 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.263694 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261729 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261733 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261736 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261739 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261742 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261745 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261748 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261750 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261753 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261756 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261759 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261761 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261764 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261766 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261769 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261772 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261775 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261778 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261781 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261784 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.264154 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261786 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261789 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261792 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261794 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261797 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261800 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261802 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261805 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261807 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261809 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261812 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261816 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261819 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261823 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261826 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261829 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261832 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261834 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261837 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261839 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.264646 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261842 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261844 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261846 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261849 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261851 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261853 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261856 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261865 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261868 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261871 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261873 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261876 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261879 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261881 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261884 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261886 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261888 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261891 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261895 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261898 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.265143 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261901 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261903 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261906 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261908 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261911 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261915 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261918 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261921 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261923 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261926 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261928 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261931 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261933 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261936 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261938 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261941 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261943 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261947 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261949 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.265632 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261952 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261955 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261958 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261961 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261963 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261966 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261969 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261971 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261986 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261989 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261991 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261994 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261996 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.261999 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262002 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262004 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262007 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262009 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262083 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262090 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262097 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:24:04.266096 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262102 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262106 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262109 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262114 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262118 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262121 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262124 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262128 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262131 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262134 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262137 2576 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262140 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262143 2576 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262147 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262151 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262155 2576 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262158 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262161 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262164 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262168 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262171 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262174 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262177 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:24:04.266589 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262181 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262184 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262187 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262190 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262193 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262195 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262201 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262204 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262207 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262210 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262213 2576 flags.go:64] FLAG: --enable-server="true" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262216 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262221 2576 flags.go:64] FLAG: --event-burst="100" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262224 2576 flags.go:64] FLAG: --event-qps="50" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262227 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262230 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262233 2576 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262236 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262239 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262242 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262245 2576 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262248 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262251 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262255 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262258 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:24:04.267142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262261 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262263 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262266 2576 flags.go:64] FLAG: --feature-gates="" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262270 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262273 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262276 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262279 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262282 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262285 2576 flags.go:64] FLAG: --help="false" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262287 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262290 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262293 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262296 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262301 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262304 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262307 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262310 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262313 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262315 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262318 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262322 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262324 2576 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262327 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262331 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:24:04.267787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262334 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262337 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262340 2576 flags.go:64] FLAG: --lock-file="" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262342 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262345 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262348 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262354 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262357 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262360 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262363 2576 flags.go:64] FLAG: --logging-format="text" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262365 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262368 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262371 2576 flags.go:64] FLAG: --manifest-url="" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262374 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262378 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262381 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262385 2576 flags.go:64] FLAG: --max-pods="110" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262388 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262391 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262394 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262397 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262401 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262404 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262407 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262414 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:24:04.268372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262417 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262420 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262423 2576 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262426 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262431 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262434 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262437 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262440 2576 flags.go:64] FLAG: --port="10250" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262443 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262446 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0233333f24b731e81" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262449 2576 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262452 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262455 2576 flags.go:64] FLAG: --register-node="true" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262458 2576 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262462 2576 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262466 2576 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262469 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262471 2576 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262475 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262478 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262481 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262484 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262488 2576 flags.go:64] FLAG: --runonce="false" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262491 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262494 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:24:04.268969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262497 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262500 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262503 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262506 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262511 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262514 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262516 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262519 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262522 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262525 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262528 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262530 2576 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262533 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262539 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262542 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262545 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262549 2576 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262552 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262555 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262558 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262561 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262564 2576 flags.go:64] FLAG: --v="2" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262567 2576 flags.go:64] FLAG: --version="false" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262572 2576 flags.go:64] FLAG: --vmodule="" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262575 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:24:04.269584 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.262578 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262695 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262698 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262701 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262705 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262708 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262711 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262713 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262716 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262718 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262721 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262725 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262728 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262730 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262733 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262736 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262739 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262741 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262745 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.270194 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262749 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262752 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262755 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262758 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262760 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262763 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262765 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262768 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262770 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262772 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262775 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262778 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262780 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262783 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262786 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262788 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262791 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262794 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262797 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262799 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.270652 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262801 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262804 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262807 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262809 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262813 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262815 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262818 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262821 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262825 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262828 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262831 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262834 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262837 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262840 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262842 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262845 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262847 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262850 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262853 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262855 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.271160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262858 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262860 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262863 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262865 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262868 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262871 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262873 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262876 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262878 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262882 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262885 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262887 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262890 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262892 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262895 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262897 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262901 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262903 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262908 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.271681 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262911 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262914 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262917 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262920 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262922 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262925 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262928 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262931 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262934 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262936 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262939 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262941 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262943 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262946 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262948 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262951 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262953 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262956 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262958 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262961 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.272162 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262964 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.262967 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.263609 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.270411 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.270426 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.270446 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.270452 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270506 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270511 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270515 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270518 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270520 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270523 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270526 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.272644 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270529 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270531 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270534 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270537 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270540 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270542 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270545 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270547 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270550 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270552 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270555 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270557 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270560 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270562 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270565 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270567 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270570 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270574 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270578 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.273107 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270581 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270585 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270588 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270592 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270595 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270598 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270601 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270604 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270607 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270611 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270614 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270617 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270619 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270622 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270624 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270627 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270629 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270632 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270634 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270637 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.273582 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270640 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270642 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270645 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270647 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270651 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270654 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270658 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270660 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270663 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270666 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270669 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270671 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270674 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270677 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270679 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270682 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270685 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270688 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270691 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.274083 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270693 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270696 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270698 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270701 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270704 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270706 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270708 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270711 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270713 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270716 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270719 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270721 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270724 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270726 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270728 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270731 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270734 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270736 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270739 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270741 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.274554 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270744 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270746 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270749 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270751 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270754 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270756 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270759 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270761 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270764 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270767 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270770 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270773 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270775 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270778 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.270783 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:24:04.275167 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270906 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270911 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270913 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270916 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270919 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270921 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270924 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270926 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270929 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270931 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270934 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270936 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270939 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270941 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270944 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270947 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270949 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270951 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270954 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270956 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:24:04.275535 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270959 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270961 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270963 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270966 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270969 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270985 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270990 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270993 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270997 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.270999 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271002 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271005 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271007 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271010 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271012 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271015 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271017 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271020 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271022 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271025 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:24:04.276027 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271027 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271030 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271032 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271034 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271037 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271040 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271043 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271045 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271047 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271050 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271052 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271055 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271057 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271060 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271062 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271065 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271068 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271070 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271073 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271076 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:24:04.276524 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271078 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271082 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271084 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271087 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271090 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271093 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271098 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271101 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271103 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271107 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271110 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271113 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271116 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271118 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271121 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271124 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271127 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271129 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271131 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:24:04.277016 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271134 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271136 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271139 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271141 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271144 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271148 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271151 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271154 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271157 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271160 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271163 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271167 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271170 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271173 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271176 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271179 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271182 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271185 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271188 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:24:04.277472 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:04.271191 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.271195 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.271302 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.273569 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.275124 2576 server.go:1019] "Starting client certificate rotation" Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.275225 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:24:04.277946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.275260 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:24:04.278125 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.277956 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:24:04.284810 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.284787 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:24:04.297462 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.297426 2576 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:24:04.299532 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.299519 2576 log.go:25] "Validated CRI v1 image API" Jul 06 22:24:04.301067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.301050 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:24:04.302287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.302270 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:24:04.303730 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.303711 2576 fs.go:135] Filesystem UUIDs: map[401028bd-38cc-4bb5-8f45-b847da5d26b1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d61337f1-58f6-4704-bb0b-02d93d671b56:/dev/nvme0n1p4] Jul 06 22:24:04.303789 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.303729 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:24:04.309133 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.309010 2576 manager.go:216] Machine: {Timestamp:2026-07-06 22:24:04.307553409 +0000 UTC m=+0.333967764 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106237 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f1d13d8d7b6c328e7e5a657f1132d SystemUUID:ec2f1d13-d8d7-b6c3-28e7-e5a657f1132d BootID:f23a8c4a-8b2b-461c-bd5a-8a3c9c442111 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:10:10:7d:4e:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:10:10:7d:4e:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:f3:3b:14:fe:8c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:24:04.309133 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.309129 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:24:04.309246 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.309218 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:24:04.311811 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.311780 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:24:04.311954 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.311815 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:24:04.312592 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.312579 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:24:04.312628 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.312594 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:24:04.312628 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.312602 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:24:04.313490 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.313476 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:24:04.314986 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.314959 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:24:04.318182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318167 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:24:04.318498 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318487 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:24:04.318533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318522 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:24:04.318561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318533 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:24:04.318561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318544 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:24:04.318561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318554 2576 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:24:04.318643 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318563 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:24:04.318955 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318943 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:24:04.319041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.318963 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:24:04.320116 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.320094 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fcbq8" Jul 06 22:24:04.322833 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.322803 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:24:04.325175 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.325153 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:24:04.325278 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.325193 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:24:04.326479 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326463 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326485 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326495 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326504 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326517 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326526 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326539 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326547 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:24:04.326561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326558 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:24:04.326798 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326566 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:24:04.326798 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326580 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:24:04.326798 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.326589 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:24:04.327555 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.327543 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:24:04.327606 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.327558 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:24:04.328093 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.328074 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fcbq8" Jul 06 22:24:04.329110 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.329084 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:24:04.329110 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.329089 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:24:04.331151 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.331135 2576 server.go:1268] "Started kubelet" Jul 06 22:24:04.331271 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.331214 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:24:04.331318 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.331299 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:24:04.331385 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.331342 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:24:04.332222 ip-10-0-139-147 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:24:04.333281 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.333261 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:24:04.334507 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.334490 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:24:04.339086 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.339047 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:24:04.339677 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.339658 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:24:04.339776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.339706 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:24:04.339846 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.339821 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.340329 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340307 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:24:04.340329 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340329 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:24:04.340485 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340306 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:24:04.340535 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340520 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:24:04.340535 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340530 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:24:04.340631 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340605 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:24:04.340631 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340616 2576 factory.go:55] Registering systemd factory Jul 06 22:24:04.340631 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340623 2576 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:24:04.340843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340821 2576 factory.go:153] Registering CRI-O factory Jul 06 22:24:04.340843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340842 2576 factory.go:223] Registration of the crio container factory successfully Jul 06 22:24:04.341018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340865 2576 factory.go:103] Registering Raw factory Jul 06 22:24:04.341018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.340880 2576 manager.go:1196] Started watching for new ooms in manager Jul 06 22:24:04.341494 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.341368 2576 manager.go:349] Starting recovery of all containers Jul 06 22:24:04.342473 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.342434 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:04.342959 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.342939 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-147.ec2.internal" not found Jul 06 22:24:04.345547 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.345518 2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-147.ec2.internal\" not found" node="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.353196 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.353060 2576 manager.go:354] Recovery completed Jul 06 22:24:04.359218 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.359199 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.359633 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.359620 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-147.ec2.internal" not found Jul 06 22:24:04.361890 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.361874 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.361953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.361901 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.361953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.361914 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.362420 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.362409 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:24:04.362420 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.362419 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:24:04.362502 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.362435 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:24:04.362502 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.362422 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.363187 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363172 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.363251 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363197 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.363251 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363208 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.363744 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363731 2576 policy_none.go:49] "None policy: Start" Jul 06 22:24:04.363793 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363752 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:24:04.363793 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.363766 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:24:04.364787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.364774 2576 policy_none.go:47] "Start" Jul 06 22:24:04.403596 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.403578 2576 manager.go:337] "Starting Device Plugin manager" Jul 06 22:24:04.403701 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.403613 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:24:04.403701 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.403623 2576 server.go:87] "Starting device plugin registration server" Jul 06 22:24:04.403905 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.403893 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:24:04.403989 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.403905 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:24:04.404061 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.404014 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:24:04.404106 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.404095 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:24:04.404106 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.404103 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:24:04.404713 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.404694 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:24:04.404805 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.404749 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.418486 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.418456 2576 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-139-147.ec2.internal" not found Jul 06 22:24:04.504288 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.504202 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.505743 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.505405 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.505743 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.505433 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.505743 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.505442 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.505743 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.505478 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.514022 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.513969 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:24:04.515267 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.515239 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:24:04.515377 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.515286 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.515377 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.515296 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:24:04.515377 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.515311 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-147.ec2.internal\": node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.515377 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.515328 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:24:04.515377 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.515368 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 22:24:04.517447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.517427 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:04.558711 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.558684 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.615761 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.615725 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal"] Jul 06 22:24:04.615859 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.615806 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.616813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.616798 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.616884 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.616828 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.616884 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.616837 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.618231 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.618217 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.618348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.618334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.618384 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.618369 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.619036 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619020 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.619114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619052 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.619114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619020 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.619114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619087 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.619114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619098 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.619114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.619064 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.620361 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.620345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.620432 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.620382 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:24:04.621151 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.621132 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:24:04.621250 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.621164 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:24:04.621250 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.621178 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:24:04.636652 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.636625 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-147.ec2.internal\" not found" node="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.641018 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.640998 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-147.ec2.internal\" not found" node="ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.641660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.641643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.641721 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.641668 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.641721 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.641688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca8eaf696cc1052bc992b005a70346d1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-147.ec2.internal\" (UID: \"ca8eaf696cc1052bc992b005a70346d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.659610 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.659581 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.741842 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.741948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.741948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.741948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca8eaf696cc1052bc992b005a70346d1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-147.ec2.internal\" (UID: \"ca8eaf696cc1052bc992b005a70346d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.741948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aabd53032737be4205f0ae99d377ffca-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal\" (UID: \"aabd53032737be4205f0ae99d377ffca\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.741948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.741939 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca8eaf696cc1052bc992b005a70346d1-config\") pod \"kube-apiserver-proxy-ip-10-0-139-147.ec2.internal\" (UID: \"ca8eaf696cc1052bc992b005a70346d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.760012 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.759920 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.860764 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.860721 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:04.939944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.939917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.945206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:04.945186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:04.961158 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:04.961119 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.061840 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.061737 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.162266 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.162231 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.262764 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.262726 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.274958 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.274941 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:24:05.275124 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.275102 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:24:05.275182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.275115 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:24:05.333492 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.333401 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:19:04 +0000 UTC" deadline="2027-12-15 09:53:40.082447714 +0000 UTC" Jul 06 22:24:05.333492 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.333443 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12635h29m34.749008138s" Jul 06 22:24:05.362880 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.362852 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.463575 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.463545 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.468952 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:05.468922 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8eaf696cc1052bc992b005a70346d1.slice/crio-b4b5580367fdb41bcb34c3860258a45c1b602e2e2b730b00deb5c756e8b78682 WatchSource:0}: Error finding container b4b5580367fdb41bcb34c3860258a45c1b602e2e2b730b00deb5c756e8b78682: Status 404 returned error can't find the container with id b4b5580367fdb41bcb34c3860258a45c1b602e2e2b730b00deb5c756e8b78682 Jul 06 22:24:05.469210 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:05.469174 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabd53032737be4205f0ae99d377ffca.slice/crio-190bcbd7e48f149b6974de049568b6278f1892cb2e3acb52781f56b3326b161e WatchSource:0}: Error finding container 190bcbd7e48f149b6974de049568b6278f1892cb2e3acb52781f56b3326b161e: Status 404 returned error can't find the container with id 190bcbd7e48f149b6974de049568b6278f1892cb2e3acb52781f56b3326b161e Jul 06 22:24:05.473661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.473643 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:24:05.518441 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.518380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" event={"ID":"aabd53032737be4205f0ae99d377ffca","Type":"ContainerStarted","Data":"190bcbd7e48f149b6974de049568b6278f1892cb2e3acb52781f56b3326b161e"} Jul 06 22:24:05.519366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.519342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" event={"ID":"ca8eaf696cc1052bc992b005a70346d1","Type":"ContainerStarted","Data":"b4b5580367fdb41bcb34c3860258a45c1b602e2e2b730b00deb5c756e8b78682"} Jul 06 22:24:05.563838 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:05.563809 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-147.ec2.internal\" not found" Jul 06 22:24:05.661843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.661814 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:05.740015 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.739964 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" Jul 06 22:24:05.752333 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.752307 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:24:05.754348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.754332 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" Jul 06 22:24:05.760587 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:05.760571 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:24:06.281593 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.281563 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:06.320078 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.320040 2576 apiserver.go:52] "Watching apiserver" Jul 06 22:24:06.328779 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.328749 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:24:06.329152 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.329131 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal","openshift-multus/network-metrics-daemon-v2h59","openshift-network-operator/iptables-alerter-wvj58","kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs","openshift-dns/node-resolver-xq8p6","openshift-image-registry/node-ca-xhdwp","openshift-multus/multus-additional-cni-plugins-j29rx","openshift-multus/multus-l75jp","openshift-network-diagnostics/network-check-target-z7wf2","openshift-ovn-kubernetes/ovnkube-node-m8grd","kube-system/global-pull-secret-syncer-plfpg","kube-system/konnectivity-agent-t6bsh","openshift-cluster-node-tuning-operator/tuned-69hk5"] Jul 06 22:24:06.330804 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.330782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:06.330912 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.330866 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:06.331913 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.331894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.333318 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.333297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.334630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.334265 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:24:06.334630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.334310 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4dsk6\"" Jul 06 22:24:06.334630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.334501 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:24:06.334630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.334512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.335519 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.335481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.335519 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.335481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.335686 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.335584 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:24:06.335900 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.335882 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.336231 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.336093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rrq5j\"" Jul 06 22:24:06.336516 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.336494 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.336751 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.336636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j9st7\"" Jul 06 22:24:06.336751 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.336708 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.337414 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.337398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.337839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.337822 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.338419 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.338310 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:24:06.338419 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.338328 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.338564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.338454 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sbkd6\"" Jul 06 22:24:06.338750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.338727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.339315 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.339299 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.339603 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.339548 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.339900 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.339883 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7dj6s\"" Jul 06 22:24:06.340246 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.340229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.340670 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.340657 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:24:06.340901 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.340882 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:24:06.341214 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.341199 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.341335 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.341275 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f54vl\"" Jul 06 22:24:06.341335 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.341323 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.341544 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.341376 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:24:06.341729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.341715 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.341803 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.341782 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:06.342305 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.342288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.342525 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.342504 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.342607 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.342536 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:24:06.343135 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.343117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-npsdl\"" Jul 06 22:24:06.343449 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.343434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.344970 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.344953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.345648 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.345627 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:24:06.345747 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.345727 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p7z6f\"" Jul 06 22:24:06.345809 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.345745 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:24:06.346431 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.346411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.346518 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.346490 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:06.347569 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347547 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:24:06.347668 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:24:06.347735 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347678 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:24:06.347786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347750 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rx4cr\"" Jul 06 22:24:06.348058 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347877 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:24:06.348058 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347915 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:24:06.348058 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.347937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:24:06.350913 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.350894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-konnectivity-ca\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.351067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.350919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-iptables-alerter-script\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.351067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.350936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fdp\" (UniqueName: \"kubernetes.io/projected/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-kube-api-access-z2fdp\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.351067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.350972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysconfig\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351026 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-sys\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8479ad7-5197-44e5-b8a0-f6d111714378-hosts-file\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5k6f\" (UniqueName: \"kubernetes.io/projected/c8479ad7-5197-44e5-b8a0-f6d111714378-kube-api-access-s5k6f\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-kubernetes\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-lib-modules\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-host\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-tmp\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-system-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351243 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-k8s-cni-cncf-io\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-bin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-hostroot\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j5q\" (UniqueName: \"kubernetes.io/projected/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-kube-api-access-78j5q\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-registration-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-netns\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5z6\" (UniqueName: \"kubernetes.io/projected/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-kube-api-access-hr5z6\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-systemd\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.351646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cni-binary-copy\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-multus\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlc9\" (UniqueName: \"kubernetes.io/projected/9a39f0e7-e0e5-4107-afcb-86938748c51b-kube-api-access-wwlc9\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8pfk\" (UniqueName: \"kubernetes.io/projected/7558251c-6816-42da-bf6a-d2f21afec92b-kube-api-access-m8pfk\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-device-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-os-release\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-kubelet\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-multus-certs\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-etc-kubernetes\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.351944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-agent-certs\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a39f0e7-e0e5-4107-afcb-86938748c51b-host\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-run\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-var-lib-kubelet\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-etc-tuned\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-host-slash\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-modprobe-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-socket-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352307 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-sys-fs\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfq9l\" (UniqueName: \"kubernetes.io/projected/34bff367-c9f5-466e-8f4d-aba766f14cee-kube-api-access-vfq9l\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cnibin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-conf-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a39f0e7-e0e5-4107-afcb-86938748c51b-serviceca\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-conf\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8479ad7-5197-44e5-b8a0-f6d111714378-tmp-dir\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-socket-dir-parent\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.352819 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.352533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-daemon-config\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.353817 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.353765 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:24:06.373463 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.373430 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dvgvn" Jul 06 22:24:06.374301 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.374279 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:06.381440 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.381416 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dvgvn" Jul 06 22:24:06.441672 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.441637 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:24:06.452899 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.452869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-etc-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.452899 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.452900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-kubelet\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.452934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-modprobe-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.452965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfq9l\" (UniqueName: \"kubernetes.io/projected/34bff367-c9f5-466e-8f4d-aba766f14cee-kube-api-access-vfq9l\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453002 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cnibin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-daemon-config\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a39f0e7-e0e5-4107-afcb-86938748c51b-serviceca\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.453148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cnibin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453182 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-modprobe-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8479ad7-5197-44e5-b8a0-f6d111714378-tmp-dir\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-system-cni-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-log-socket\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453365 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-conf-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-konnectivity-ca\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-iptables-alerter-script\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453493 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-conf-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-sys\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8479ad7-5197-44e5-b8a0-f6d111714378-hosts-file\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5k6f\" (UniqueName: \"kubernetes.io/projected/c8479ad7-5197-44e5-b8a0-f6d111714378-kube-api-access-s5k6f\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-hostroot\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-systemd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-kubernetes\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-host\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-tmp\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8479ad7-5197-44e5-b8a0-f6d111714378-tmp-dir\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-hostroot\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-dbus\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j5q\" (UniqueName: \"kubernetes.io/projected/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-kube-api-access-78j5q\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a39f0e7-e0e5-4107-afcb-86938748c51b-serviceca\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453835 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8479ad7-5197-44e5-b8a0-f6d111714378-hosts-file\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.453843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-node-log\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453838 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-daemon-config\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453939 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-sys\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.453945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-d\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-host\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454082 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-kubernetes\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454103 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-konnectivity-ca\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-env-overrides\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-systemd\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.454279 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454293 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-iptables-alerter-script\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-systemd\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.454357 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:06.954336232 +0000 UTC m=+2.980750599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-multus-certs\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.454464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlc9\" (UniqueName: \"kubernetes.io/projected/9a39f0e7-e0e5-4107-afcb-86938748c51b-kube-api-access-wwlc9\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8pfk\" (UniqueName: \"kubernetes.io/projected/7558251c-6816-42da-bf6a-d2f21afec92b-kube-api-access-m8pfk\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-multus-certs\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-device-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-os-release\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-device-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-agent-certs\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454717 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-os-release\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a39f0e7-e0e5-4107-afcb-86938748c51b-host\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-run\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-var-lib-kubelet\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-etc-tuned\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-run\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454865 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-var-lib-kubelet\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a39f0e7-e0e5-4107-afcb-86938748c51b-host\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7wl\" (UniqueName: \"kubernetes.io/projected/89cdd929-92a4-450e-a67a-ebc2259983ec-kube-api-access-lz7wl\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.455252 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-host-slash\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.454971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-socket-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-sys-fs\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455086 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-conf\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-socket-dir-parent\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-host-slash\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-os-release\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-multus-socket-dir-parent\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-systemd-units\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-slash\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-script-lib\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysctl-conf\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prl5\" (UniqueName: \"kubernetes.io/projected/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-kube-api-access-8prl5\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.455962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fdp\" (UniqueName: \"kubernetes.io/projected/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-kube-api-access-z2fdp\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysconfig\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455460 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-socket-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5z6\" (UniqueName: \"kubernetes.io/projected/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-kube-api-access-hr5z6\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-netns\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455506 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-sys-fs\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-ovn\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455533 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-etc-sysconfig\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-netd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-kubelet-config\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-lib-modules\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-system-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-k8s-cni-cncf-io\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-bin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7558251c-6816-42da-bf6a-d2f21afec92b-lib-modules\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455838 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-system-cni-dir\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.456762 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-bin\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-k8s-cni-cncf-io\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-var-lib-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-bin\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-config\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.455997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-registration-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-netns\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-cnibin\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456102 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-run-netns\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456103 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-registration-dir\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cni-binary-copy\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34bff367-c9f5-466e-8f4d-aba766f14cee-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-multus\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456230 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-cni-multus\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-kubelet\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.457434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-etc-kubernetes\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-host-var-lib-kubelet\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456369 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-etc-kubernetes\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.456649 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-cni-binary-copy\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.457967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-tmp\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.458121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.458022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7558251c-6816-42da-bf6a-d2f21afec92b-etc-tuned\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.458353 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.458262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1cb0358-d19d-4f0a-9bf3-22d6d864a149-agent-certs\") pod \"konnectivity-agent-t6bsh\" (UID: \"a1cb0358-d19d-4f0a-9bf3-22d6d864a149\") " pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.460579 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.460555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfq9l\" (UniqueName: \"kubernetes.io/projected/34bff367-c9f5-466e-8f4d-aba766f14cee-kube-api-access-vfq9l\") pod \"aws-ebs-csi-driver-node-kwxxs\" (UID: \"34bff367-c9f5-466e-8f4d-aba766f14cee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.461885 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.461763 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:06.461885 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.461788 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:06.461885 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.461802 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:06.461885 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.461882 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:06.961861979 +0000 UTC m=+2.988276341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:06.462346 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.462320 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j5q\" (UniqueName: \"kubernetes.io/projected/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-kube-api-access-78j5q\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.463595 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.463573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5k6f\" (UniqueName: \"kubernetes.io/projected/c8479ad7-5197-44e5-b8a0-f6d111714378-kube-api-access-s5k6f\") pod \"node-resolver-xq8p6\" (UID: \"c8479ad7-5197-44e5-b8a0-f6d111714378\") " pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.464192 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.464172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlc9\" (UniqueName: \"kubernetes.io/projected/9a39f0e7-e0e5-4107-afcb-86938748c51b-kube-api-access-wwlc9\") pod \"node-ca-xhdwp\" (UID: \"9a39f0e7-e0e5-4107-afcb-86938748c51b\") " pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.464443 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.464416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5z6\" (UniqueName: \"kubernetes.io/projected/e8067716-03ba-4d64-bfd8-ed81e0aa6ff4-kube-api-access-hr5z6\") pod \"multus-l75jp\" (UID: \"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4\") " pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.464779 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.464760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8pfk\" (UniqueName: \"kubernetes.io/projected/7558251c-6816-42da-bf6a-d2f21afec92b-kube-api-access-m8pfk\") pod \"tuned-69hk5\" (UID: \"7558251c-6816-42da-bf6a-d2f21afec92b\") " pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.465044 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.465028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fdp\" (UniqueName: \"kubernetes.io/projected/79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc-kube-api-access-z2fdp\") pod \"iptables-alerter-wvj58\" (UID: \"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc\") " pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.557682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.557682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-etc-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.557682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-kubelet\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-etc-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557801 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-kubelet\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557856 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-system-cni-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-system-cni-dir\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-log-socket\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.557944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.557962 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.557997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-systemd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-log-socket\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.558082 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:07.05805878 +0000 UTC m=+3.084473131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-systemd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-dbus\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-node-log\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-env-overrides\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-node-log\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7wl\" (UniqueName: \"kubernetes.io/projected/89cdd929-92a4-450e-a67a-ebc2259983ec-kube-api-access-lz7wl\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558341 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-dbus\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.558417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-os-release\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-systemd-units\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-os-release\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-systemd-units\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-env-overrides\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-slash\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-slash\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558737 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-script-lib\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prl5\" (UniqueName: \"kubernetes.io/projected/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-kube-api-access-8prl5\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-netns\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-ovn\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-netd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558863 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-run-netns\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-kubelet-config\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.559206 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b3f4825-2adc-4709-b51c-1cdb45711326-kubelet-config\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-var-lib-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-run-ovn\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.558990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-bin\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-config\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559020 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-netd\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-cnibin\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-var-lib-openvswitch\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559192 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89cdd929-92a4-450e-a67a-ebc2259983ec-cnibin\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-script-lib\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559231 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-host-cni-bin\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.559939 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.559525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovnkube-config\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.560392 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.560033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89cdd929-92a4-450e-a67a-ebc2259983ec-cni-binary-copy\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.561082 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.561064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-ovn-node-metrics-cert\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.566644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.566616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7wl\" (UniqueName: \"kubernetes.io/projected/89cdd929-92a4-450e-a67a-ebc2259983ec-kube-api-access-lz7wl\") pod \"multus-additional-cni-plugins-j29rx\" (UID: \"89cdd929-92a4-450e-a67a-ebc2259983ec\") " pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.567418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.567397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prl5\" (UniqueName: \"kubernetes.io/projected/2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f-kube-api-access-8prl5\") pod \"ovnkube-node-m8grd\" (UID: \"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.651799 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.651765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:06.658100 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.658067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wvj58" Jul 06 22:24:06.667009 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.666972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-69hk5" Jul 06 22:24:06.669481 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.669461 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:06.672416 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.672391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" Jul 06 22:24:06.679129 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.679105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xq8p6" Jul 06 22:24:06.685962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.685939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l75jp" Jul 06 22:24:06.692873 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.692844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xhdwp" Jul 06 22:24:06.698606 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.698581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j29rx" Jul 06 22:24:06.704577 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.704553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:06.961848 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:06.961816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:06.962032 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.961946 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:06.962090 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:06.962048 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:07.962028026 +0000 UTC m=+3.988442371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:07.062850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.062808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:07.063015 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.062869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:07.063015 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.062965 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:07.063015 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.062987 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:07.063015 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.063011 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:07.063175 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.063021 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:07.063175 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.063041 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:08.063023488 +0000 UTC m=+4.089437836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:07.063175 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.063065 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:08.063051838 +0000 UTC m=+4.089466180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:07.331160 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.331132 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8067716_03ba_4d64_bfd8_ed81e0aa6ff4.slice/crio-75b7e28d778131679453818d8ad3aeec4be18dca30dc15ac67a530fafb1d8c92 WatchSource:0}: Error finding container 75b7e28d778131679453818d8ad3aeec4be18dca30dc15ac67a530fafb1d8c92: Status 404 returned error can't find the container with id 75b7e28d778131679453818d8ad3aeec4be18dca30dc15ac67a530fafb1d8c92 Jul 06 22:24:07.333867 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.333763 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7558251c_6816_42da_bf6a_d2f21afec92b.slice/crio-82fc24a2496004945fe9bf4ebc6b84aa44dea090ae581a30fa0ebee93b26a820 WatchSource:0}: Error finding container 82fc24a2496004945fe9bf4ebc6b84aa44dea090ae581a30fa0ebee93b26a820: Status 404 returned error can't find the container with id 82fc24a2496004945fe9bf4ebc6b84aa44dea090ae581a30fa0ebee93b26a820 Jul 06 22:24:07.337056 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.337030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f36b3d_75aa_4dd9_a0bb_a4c641cb21fc.slice/crio-5b140782a2f36176878379bd764b7d971fa128d06ba861f8b5a0f4df7fa50768 WatchSource:0}: Error finding container 5b140782a2f36176878379bd764b7d971fa128d06ba861f8b5a0f4df7fa50768: Status 404 returned error can't find the container with id 5b140782a2f36176878379bd764b7d971fa128d06ba861f8b5a0f4df7fa50768 Jul 06 22:24:07.338321 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.338295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cb0358_d19d_4f0a_9bf3_22d6d864a149.slice/crio-07306ba5613b9d6ae8b4aa2e6240a7ad2e53006a03a5c3721b454310267338fc WatchSource:0}: Error finding container 07306ba5613b9d6ae8b4aa2e6240a7ad2e53006a03a5c3721b454310267338fc: Status 404 returned error can't find the container with id 07306ba5613b9d6ae8b4aa2e6240a7ad2e53006a03a5c3721b454310267338fc Jul 06 22:24:07.339558 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.339534 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cdd929_92a4_450e_a67a_ebc2259983ec.slice/crio-c21f5f80f092a2cdb9eca86568df4bec6cd24935f7c87092de4cfc1beed71bd2 WatchSource:0}: Error finding container c21f5f80f092a2cdb9eca86568df4bec6cd24935f7c87092de4cfc1beed71bd2: Status 404 returned error can't find the container with id c21f5f80f092a2cdb9eca86568df4bec6cd24935f7c87092de4cfc1beed71bd2 Jul 06 22:24:07.340465 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.340437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bff367_c9f5_466e_8f4d_aba766f14cee.slice/crio-88d93ebba8a1dd5ead6cac00505f4db20f3631d7ea80fda158a754d3a506087a WatchSource:0}: Error finding container 88d93ebba8a1dd5ead6cac00505f4db20f3631d7ea80fda158a754d3a506087a: Status 404 returned error can't find the container with id 88d93ebba8a1dd5ead6cac00505f4db20f3631d7ea80fda158a754d3a506087a Jul 06 22:24:07.342048 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.342021 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8479ad7_5197_44e5_b8a0_f6d111714378.slice/crio-dbb2a33164381a1a3d5c07b7a47b9f5b9e4d2f1dc0c20caeec9fd26a67587270 WatchSource:0}: Error finding container dbb2a33164381a1a3d5c07b7a47b9f5b9e4d2f1dc0c20caeec9fd26a67587270: Status 404 returned error can't find the container with id dbb2a33164381a1a3d5c07b7a47b9f5b9e4d2f1dc0c20caeec9fd26a67587270 Jul 06 22:24:07.343304 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.343208 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8fbb95_4ee0_42ef_94e8_c1e4d3048f2f.slice/crio-0bb9304934f9d4a875fe4a301153b268f4f1be57a88537776225aedb1fdd036c WatchSource:0}: Error finding container 0bb9304934f9d4a875fe4a301153b268f4f1be57a88537776225aedb1fdd036c: Status 404 returned error can't find the container with id 0bb9304934f9d4a875fe4a301153b268f4f1be57a88537776225aedb1fdd036c Jul 06 22:24:07.345816 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:07.345573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a39f0e7_e0e5_4107_afcb_86938748c51b.slice/crio-4b74916355df04156c06a85b08db53ad60ace34316acf58950e43a265e225e12 WatchSource:0}: Error finding container 4b74916355df04156c06a85b08db53ad60ace34316acf58950e43a265e225e12: Status 404 returned error can't find the container with id 4b74916355df04156c06a85b08db53ad60ace34316acf58950e43a265e225e12 Jul 06 22:24:07.384448 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.384280 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:19:06 +0000 UTC" deadline="2027-12-01 01:48:11.482237727 +0000 UTC" Jul 06 22:24:07.384448 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.384444 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12291h24m4.097797476s" Jul 06 22:24:07.516105 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.516075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:07.516267 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.516192 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:07.524683 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.524648 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerStarted","Data":"c21f5f80f092a2cdb9eca86568df4bec6cd24935f7c87092de4cfc1beed71bd2"} Jul 06 22:24:07.525835 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.525797 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69hk5" event={"ID":"7558251c-6816-42da-bf6a-d2f21afec92b","Type":"ContainerStarted","Data":"82fc24a2496004945fe9bf4ebc6b84aa44dea090ae581a30fa0ebee93b26a820"} Jul 06 22:24:07.527231 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.527207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l75jp" event={"ID":"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4","Type":"ContainerStarted","Data":"75b7e28d778131679453818d8ad3aeec4be18dca30dc15ac67a530fafb1d8c92"} Jul 06 22:24:07.528383 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.528358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"0bb9304934f9d4a875fe4a301153b268f4f1be57a88537776225aedb1fdd036c"} Jul 06 22:24:07.529382 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.529358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6bsh" event={"ID":"a1cb0358-d19d-4f0a-9bf3-22d6d864a149","Type":"ContainerStarted","Data":"07306ba5613b9d6ae8b4aa2e6240a7ad2e53006a03a5c3721b454310267338fc"} Jul 06 22:24:07.531243 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.531223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wvj58" event={"ID":"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc","Type":"ContainerStarted","Data":"5b140782a2f36176878379bd764b7d971fa128d06ba861f8b5a0f4df7fa50768"} Jul 06 22:24:07.533243 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.533209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" event={"ID":"ca8eaf696cc1052bc992b005a70346d1","Type":"ContainerStarted","Data":"2e6b553c98383fc2d17cc896d6cc76016c3a86d6bd433f55668a2bef7b2e56e2"} Jul 06 22:24:07.534463 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.534439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xhdwp" event={"ID":"9a39f0e7-e0e5-4107-afcb-86938748c51b","Type":"ContainerStarted","Data":"4b74916355df04156c06a85b08db53ad60ace34316acf58950e43a265e225e12"} Jul 06 22:24:07.535582 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.535555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xq8p6" event={"ID":"c8479ad7-5197-44e5-b8a0-f6d111714378","Type":"ContainerStarted","Data":"dbb2a33164381a1a3d5c07b7a47b9f5b9e4d2f1dc0c20caeec9fd26a67587270"} Jul 06 22:24:07.536795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.536762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" event={"ID":"34bff367-c9f5-466e-8f4d-aba766f14cee","Type":"ContainerStarted","Data":"88d93ebba8a1dd5ead6cac00505f4db20f3631d7ea80fda158a754d3a506087a"} Jul 06 22:24:07.547633 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.547580 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-147.ec2.internal" podStartSLOduration=2.547565615 podStartE2EDuration="2.547565615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:07.547093718 +0000 UTC m=+3.573508083" watchObservedRunningTime="2026-07-06 22:24:07.547565615 +0000 UTC m=+3.573979978" Jul 06 22:24:07.968812 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:07.968768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:07.969050 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.968994 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:07.969115 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:07.969060 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:09.969041907 +0000 UTC m=+5.995456248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:08.070233 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.070202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:08.070410 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.070277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:08.070482 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.070422 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:08.070527 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.070487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:10.070468452 +0000 UTC m=+6.096882811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:08.070944 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.070924 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:08.071100 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.070951 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:08.071100 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.070964 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:08.071100 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.071024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:10.071008678 +0000 UTC m=+6.097423022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:08.385254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.385186 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:19:06 +0000 UTC" deadline="2028-03-28 14:07:09.950100438 +0000 UTC" Jul 06 22:24:08.385254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.385252 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15135h43m1.564851777s" Jul 06 22:24:08.535000 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.534724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:08.535000 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.534852 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:08.535000 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.534914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:08.535328 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:08.535108 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:08.545387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.544559 2576 generic.go:356] "Generic (PLEG): container finished" podID="aabd53032737be4205f0ae99d377ffca" containerID="7db9590b502b9a5231e42214c1987b5f58693fc9a2f76765377c6042b7ed542a" exitCode=0 Jul 06 22:24:08.545387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:08.545093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" event={"ID":"aabd53032737be4205f0ae99d377ffca","Type":"ContainerDied","Data":"7db9590b502b9a5231e42214c1987b5f58693fc9a2f76765377c6042b7ed542a"} Jul 06 22:24:09.516273 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:09.516242 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:09.516874 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:09.516383 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:09.566360 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:09.566325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" event={"ID":"aabd53032737be4205f0ae99d377ffca","Type":"ContainerStarted","Data":"2835ae3d85cdaec5710498220cdb572337a522277c924d0b7e6ea667b6392643"} Jul 06 22:24:09.993217 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:09.993180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:09.993416 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:09.993384 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:09.993501 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:09.993461 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:13.993432872 +0000 UTC m=+10.019847218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:10.094199 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:10.094162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:10.094380 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:10.094245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:10.094380 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094326 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:10.094380 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094372 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:10.094536 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094389 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:10.094536 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094401 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:14.094381298 +0000 UTC m=+10.120795640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:10.094536 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094402 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:10.094536 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.094447 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:14.094436672 +0000 UTC m=+10.120851030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:10.516221 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:10.516180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:10.516566 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.516322 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:10.516566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:10.516180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:10.516566 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:10.516530 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:11.516485 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:11.515986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:11.516485 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:11.516131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:12.515613 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:12.515585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:12.515781 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:12.515687 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:12.516174 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:12.516149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:12.516280 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:12.516254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:13.515835 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:13.515801 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:13.516278 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:13.515969 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:14.026922 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:14.026879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:14.027137 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.027117 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:14.027197 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.027186 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:22.027166318 +0000 UTC m=+18.053580664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:14.127601 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:14.127676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.127847 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.127867 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.127881 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.127942 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:22.12792349 +0000 UTC m=+18.154337833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.128199 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:14.128307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.128273 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:22.128241734 +0000 UTC m=+18.154656078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:14.517864 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:14.517248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:14.517864 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.517366 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:14.517864 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:14.517434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:14.517864 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:14.517503 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:15.516323 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:15.516112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:15.516323 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:15.516275 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:16.516042 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:16.516000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:16.516510 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:16.516141 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:16.516742 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:16.516609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:16.516742 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:16.516720 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:17.515837 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:17.515796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:17.516041 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:17.515946 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:18.515634 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:18.515592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:18.515634 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:18.515641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:18.516099 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:18.515730 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:18.516099 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:18.515833 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:19.516192 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:19.516150 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:19.516650 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:19.516312 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:20.516648 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:20.516613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:20.517055 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:20.516727 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:20.517055 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:20.516768 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:20.517055 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:20.516837 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:21.515931 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:21.515894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:21.516167 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:21.516044 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:22.091748 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:22.091719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:22.092137 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.091836 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:22.092137 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.091887 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:24:38.091873359 +0000 UTC m=+34.118287702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:22.192167 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:22.192139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:22.192184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192263 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192283 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192302 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192310 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:38.192296354 +0000 UTC m=+34.218710696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192311 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:22.192340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.192338 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:38.192332995 +0000 UTC m=+34.218747337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:22.519521 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:22.519487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:22.519737 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.519597 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:22.519737 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:22.519628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:22.519737 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:22.519697 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:23.515668 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:23.515634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:23.516103 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:23.515761 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:24.517110 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:24.517071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:24.517566 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:24.517184 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:24.517566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:24.517266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:24.517566 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:24.517399 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:25.516204 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:25.516163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:25.516402 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:25.516317 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:26.516583 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.516296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:26.517244 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.516342 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:26.517244 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:26.516703 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:26.517244 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:26.516839 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:26.605173 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.605144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xhdwp" event={"ID":"9a39f0e7-e0e5-4107-afcb-86938748c51b","Type":"ContainerStarted","Data":"4a73af1ca2be09fa8263678bc1b419bf2e05a41796a085f75d3416b78509f1f7"} Jul 06 22:24:26.606581 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.606559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xq8p6" event={"ID":"c8479ad7-5197-44e5-b8a0-f6d111714378","Type":"ContainerStarted","Data":"8c870fbfc08bd965cb5351b2e6d105af042b2bab0f8a40ba7cbe4b8a60c16e09"} Jul 06 22:24:26.607910 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.607889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" event={"ID":"34bff367-c9f5-466e-8f4d-aba766f14cee","Type":"ContainerStarted","Data":"62797f047258d133f0cd66c66cb96ba8a715885d89ed195a5cb199e95dba7a97"} Jul 06 22:24:26.609393 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.609342 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="683b1e57f73d144d1b423967769c2f267aea029561b437a1cd558dc3b849a524" exitCode=0 Jul 06 22:24:26.609483 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.609400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"683b1e57f73d144d1b423967769c2f267aea029561b437a1cd558dc3b849a524"} Jul 06 22:24:26.610900 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.610864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-69hk5" event={"ID":"7558251c-6816-42da-bf6a-d2f21afec92b","Type":"ContainerStarted","Data":"d4f7a39712d26f7bd01a38009c84dd5a5ac0ceea277a7635c85793556ff62bce"} Jul 06 22:24:26.612262 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.612240 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l75jp" event={"ID":"e8067716-03ba-4d64-bfd8-ed81e0aa6ff4","Type":"ContainerStarted","Data":"8553ff4a49ce6080fe60a00a95edc772132af8a6b4556c7a652e4468ff437fb3"} Jul 06 22:24:26.614191 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.614169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"652cead086d8272e750640b817c7a13333cebf1a523dcbbcb5e0c373260ac3eb"} Jul 06 22:24:26.614291 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.614222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"f9121620ca610461865ed8ec1b9e06e47f7c28d51310190d1eedd14fb764baef"} Jul 06 22:24:26.614291 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.614261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"c7abf994dade139f718b36c5b0bc8495d2880ace623a2b1e57907b6d415d987e"} Jul 06 22:24:26.615297 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.615277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6bsh" event={"ID":"a1cb0358-d19d-4f0a-9bf3-22d6d864a149","Type":"ContainerStarted","Data":"994fe152c78c742b4fc185b1dba2a3de65a2a6e9a242947746548eab475eefc7"} Jul 06 22:24:26.619881 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.619843 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-147.ec2.internal" podStartSLOduration=21.619831703 podStartE2EDuration="21.619831703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:09.582867913 +0000 UTC m=+5.609282278" watchObservedRunningTime="2026-07-06 22:24:26.619831703 +0000 UTC m=+22.646246067" Jul 06 22:24:26.620112 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.620091 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xhdwp" podStartSLOduration=3.858334883 podStartE2EDuration="22.620086818s" totalImagesPullingTime="18.761751935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.619364317 +0000 UTC m=+22.645778681" watchObservedRunningTime="2026-07-06 22:24:26.620086818 +0000 UTC m=+22.646501181" Jul 06 22:24:26.646402 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.646358 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t6bsh" podStartSLOduration=7.732247109 podStartE2EDuration="22.646337063s" totalImagesPullingTime="14.914089954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.632206412 +0000 UTC m=+22.658620800" watchObservedRunningTime="2026-07-06 22:24:26.646337063 +0000 UTC m=+22.672751426" Jul 06 22:24:26.646695 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.646670 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xq8p6" podStartSLOduration=3.881864118 podStartE2EDuration="22.646660521s" totalImagesPullingTime="18.764796403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.646132061 +0000 UTC m=+22.672546425" watchObservedRunningTime="2026-07-06 22:24:26.646660521 +0000 UTC m=+22.673074889" Jul 06 22:24:26.684914 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.684863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-69hk5" podStartSLOduration=3.9094454279999997 podStartE2EDuration="22.684847476s" totalImagesPullingTime="18.775402048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.684408377 +0000 UTC m=+22.710822743" watchObservedRunningTime="2026-07-06 22:24:26.684847476 +0000 UTC m=+22.711261840" Jul 06 22:24:26.701115 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:26.701074 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l75jp" podStartSLOduration=3.864340077 podStartE2EDuration="22.701056894s" totalImagesPullingTime="18.836716817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:26.700999033 +0000 UTC m=+22.727413389" watchObservedRunningTime="2026-07-06 22:24:26.701056894 +0000 UTC m=+22.727471260" Jul 06 22:24:27.516245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.516212 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:27.516515 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:27.516324 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:27.545322 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.545303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:27.623823 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.623786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"a8c7296b13385652733480dc05df76d43340c11c980a874b00a2603418cdb226"} Jul 06 22:24:27.624002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.623897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"60a3b47008f77d73e774bb062f03e0363c88cac09aa701ccc8c34181221a7eb5"} Jul 06 22:24:27.624002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.623951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"dfff660bfc2603dccbef8777ebd95664e64d1bce8112452f8096ea72a011783f"} Jul 06 22:24:27.625619 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.625547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wvj58" event={"ID":"79f36b3d-75aa-4dd9-a0bb-a4c641cb21fc","Type":"ContainerStarted","Data":"a24b955def6bd0a49899ce8d4e20bae6f211dfd64469604e6a7f64b6387733ad"} Jul 06 22:24:27.647008 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.646936 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wvj58" podStartSLOduration=4.879336402 podStartE2EDuration="23.646915293s" totalImagesPullingTime="18.767578891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:27.645862566 +0000 UTC m=+23.672276942" watchObservedRunningTime="2026-07-06 22:24:27.646915293 +0000 UTC m=+23.673329658" Jul 06 22:24:27.656057 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:27.656010 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:24:28.416519 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.416368 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:24:27.656036345Z","UUID":"9eda7c64-372c-4cf7-ab61-fa5f1f7283b3","Handler":null,"Name":"","Endpoint":""} Jul 06 22:24:28.418420 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.418398 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:24:28.418560 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.418435 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:24:28.516066 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.516035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:28.516066 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.516079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:28.516322 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:28.516171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:28.516322 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:28.516295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:28.630079 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:28.630027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" event={"ID":"34bff367-c9f5-466e-8f4d-aba766f14cee","Type":"ContainerStarted","Data":"d77d743ce982a3b98c86719f010ea1a04cfb179c11c12fd3f6b84e2da204fbb5"} Jul 06 22:24:29.216235 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.215991 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:29.216642 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.216615 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:29.515791 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.515763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:29.515964 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:29.515868 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:29.634711 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.634673 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" event={"ID":"34bff367-c9f5-466e-8f4d-aba766f14cee","Type":"ContainerStarted","Data":"cbbe4b7d86417ecf72febbded92946b49a8029f9a32875892cf06cf8234fecee"} Jul 06 22:24:29.638119 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.638089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"218e38e5393d80b7d596b3094646550afffcbe5b24a33110c7fde72e97d3c9d0"} Jul 06 22:24:29.638816 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.638803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t6bsh" Jul 06 22:24:29.651769 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:29.651723 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwxxs" podStartSLOduration=4.510858145 podStartE2EDuration="25.65170831s" totalImagesPullingTime="21.140850165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:29.65161513 +0000 UTC m=+25.678029493" watchObservedRunningTime="2026-07-06 22:24:29.65170831 +0000 UTC m=+25.678122674" Jul 06 22:24:30.516094 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:30.516058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:30.516324 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:30.516104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:30.516324 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:30.516196 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:30.516442 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:30.516321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:31.515865 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:31.515688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:31.516508 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:31.515945 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:31.644661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:31.644630 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="6ce88bfd01b87cffcb9a07c8fb9547a34bf1bf6d9ea8f841a291fc624a624e1f" exitCode=0 Jul 06 22:24:31.644843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:31.644712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"6ce88bfd01b87cffcb9a07c8fb9547a34bf1bf6d9ea8f841a291fc624a624e1f"} Jul 06 22:24:31.648329 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:31.648301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" event={"ID":"2d8fbb95-4ee0-42ef-94e8-c1e4d3048f2f","Type":"ContainerStarted","Data":"486e735779377c38105f6f23f622ec5a315739690f97fb91f9924e09c8eb7250"} Jul 06 22:24:31.695233 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:31.695135 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" podStartSLOduration=8.849554887 podStartE2EDuration="27.695120594s" totalImagesPullingTime="18.845565707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:31.694702846 +0000 UTC m=+27.721117210" watchObservedRunningTime="2026-07-06 22:24:31.695120594 +0000 UTC m=+27.721534958" Jul 06 22:24:32.516897 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.516388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:32.516897 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:32.516527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:32.517700 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.517552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:32.517700 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:32.517661 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:32.650891 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.650859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:32.651102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.650944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:32.651102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.651012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:32.667825 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.667782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:32.668011 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:32.667918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:24:33.171911 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.171876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2h59"] Jul 06 22:24:33.172160 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.172043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:33.172237 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:33.172171 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:33.172586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.172560 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plfpg"] Jul 06 22:24:33.172694 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.172667 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:33.172884 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:33.172771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:33.173436 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.173405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7wf2"] Jul 06 22:24:33.173513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.173504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:33.173613 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:33.173586 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:33.654765 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.654721 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="6fc7872dc1f05f07271fe8eabb9a151337ce8e13b3cdd6a337d68a28dafb217a" exitCode=0 Jul 06 22:24:33.655211 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:33.654799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"6fc7872dc1f05f07271fe8eabb9a151337ce8e13b3cdd6a337d68a28dafb217a"} Jul 06 22:24:34.517787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:34.517542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:34.518052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:34.517585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:34.518052 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:34.517893 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:34.518052 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:34.517954 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:34.518052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:34.517629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:34.518275 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:34.518101 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:35.662293 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:35.662253 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="74d88e4304bb71002fe0f727f5ffe46be7d9e22c25be3ebd1c66f58c20432947" exitCode=0 Jul 06 22:24:35.662770 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:35.662331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"74d88e4304bb71002fe0f727f5ffe46be7d9e22c25be3ebd1c66f58c20432947"} Jul 06 22:24:36.516401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:36.516366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:36.516590 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:36.516373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:36.516590 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:36.516493 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:36.516717 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:36.516595 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:36.516717 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:36.516378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:36.516717 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:36.516695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:38.105652 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.105610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:38.106248 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.105778 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:38.106248 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.105860 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs podName:f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd nodeName:}" failed. No retries permitted until 2026-07-06 22:25:10.105838242 +0000 UTC m=+66.132252585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs") pod "network-metrics-daemon-v2h59" (UID: "f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:38.206159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.206120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:38.206340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.206186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:38.206340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206281 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:38.206340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206308 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:38.206340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206323 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:38.206340 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206334 2576 projected.go:196] Error preparing data for projected volume kube-api-access-jplk4 for pod openshift-network-diagnostics/network-check-target-z7wf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:38.206559 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206355 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret podName:9b3f4825-2adc-4709-b51c-1cdb45711326 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:10.206335726 +0000 UTC m=+66.232750089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret") pod "global-pull-secret-syncer-plfpg" (UID: "9b3f4825-2adc-4709-b51c-1cdb45711326") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:38.206559 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.206373 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4 podName:c9fe3494-24a7-4d74-8ffd-13b52d788547 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:10.206363268 +0000 UTC m=+66.232777610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jplk4" (UniqueName: "kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4") pod "network-check-target-z7wf2" (UID: "c9fe3494-24a7-4d74-8ffd-13b52d788547") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:38.516057 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.516018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:38.516057 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.516036 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:38.516303 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.516144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:38.516303 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.516159 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-plfpg" podUID="9b3f4825-2adc-4709-b51c-1cdb45711326" Jul 06 22:24:38.516303 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.516246 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7wf2" podUID="c9fe3494-24a7-4d74-8ffd-13b52d788547" Jul 06 22:24:38.516445 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:38.516356 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2h59" podUID="f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd" Jul 06 22:24:38.762472 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.762442 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-147.ec2.internal" event="NodeReady" Jul 06 22:24:38.762671 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.762599 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:24:38.810337 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.810227 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tlk95"] Jul 06 22:24:38.837097 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.837061 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ntp85"] Jul 06 22:24:38.837280 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.837180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:38.840198 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.840169 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:24:38.840198 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.840176 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2cqxg\"" Jul 06 22:24:38.840404 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.840340 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:24:38.852953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.852920 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlk95"] Jul 06 22:24:38.853166 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.852966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ntp85"] Jul 06 22:24:38.853166 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.853023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:38.855987 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.855921 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:24:38.856153 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.856055 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:24:38.856153 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.856119 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:24:38.856272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.856265 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tzl8n\"" Jul 06 22:24:38.900929 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.900891 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp"] Jul 06 22:24:38.911270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.910991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-config-volume\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:38.911270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.911052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp6m\" (UniqueName: \"kubernetes.io/projected/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-kube-api-access-jnp6m\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:38.911270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.911123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:38.911270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.911159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-tmp-dir\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:38.922812 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.922778 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp"] Jul 06 22:24:38.923009 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.922924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" Jul 06 22:24:38.925736 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.925708 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 22:24:38.925924 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.925897 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:38.925924 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:38.925909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jmrm9\"" Jul 06 22:24:39.012297 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-config-volume\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.012515 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp6m\" (UniqueName: \"kubernetes.io/projected/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-kube-api-access-jnp6m\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.012515 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-cert\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.012515 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvmc\" (UniqueName: \"kubernetes.io/projected/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-kube-api-access-stvmc\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.012515 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljzq\" (UniqueName: \"kubernetes.io/projected/ad3264a6-1ef1-49d8-a9cf-3d479eb1466b-kube-api-access-zljzq\") pod \"migrator-d57d58cc9-bjqkp\" (UID: \"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" Jul 06 22:24:39.012726 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.012726 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-tmp-dir\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.012726 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.012646 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:39.012726 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.012705 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls podName:e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e nodeName:}" failed. No retries permitted until 2026-07-06 22:24:39.512685121 +0000 UTC m=+35.539099471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls") pod "dns-default-tlk95" (UID: "e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e") : secret "dns-default-metrics-tls" not found Jul 06 22:24:39.012987 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.012954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-tmp-dir\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.013128 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.013007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-config-volume\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.032257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.032226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp6m\" (UniqueName: \"kubernetes.io/projected/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-kube-api-access-jnp6m\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.050409 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.050360 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7m8tk"] Jul 06 22:24:39.075320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.075228 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7m8tk"] Jul 06 22:24:39.075489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.075391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.078127 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.077953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:24:39.078127 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.078110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9tvqz\"" Jul 06 22:24:39.078351 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.078203 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:24:39.078351 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.078326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:24:39.078458 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.078377 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:24:39.113118 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.113080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-cert\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.113118 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.113117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvmc\" (UniqueName: \"kubernetes.io/projected/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-kube-api-access-stvmc\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.113800 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.113136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljzq\" (UniqueName: \"kubernetes.io/projected/ad3264a6-1ef1-49d8-a9cf-3d479eb1466b-kube-api-access-zljzq\") pod \"migrator-d57d58cc9-bjqkp\" (UID: \"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" Jul 06 22:24:39.116129 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.116088 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-cert\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.122371 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.122333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljzq\" (UniqueName: \"kubernetes.io/projected/ad3264a6-1ef1-49d8-a9cf-3d479eb1466b-kube-api-access-zljzq\") pod \"migrator-d57d58cc9-bjqkp\" (UID: \"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" Jul 06 22:24:39.122516 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.122370 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvmc\" (UniqueName: \"kubernetes.io/projected/e70472f5-e8bd-4ddb-9ea4-0c4559febb51-kube-api-access-stvmc\") pod \"ingress-canary-ntp85\" (UID: \"e70472f5-e8bd-4ddb-9ea4-0c4559febb51\") " pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.164460 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.164416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ntp85" Jul 06 22:24:39.214203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.214159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.214389 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.214229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5d9\" (UniqueName: \"kubernetes.io/projected/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-api-access-bt5d9\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.214389 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.214269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c5692a4-c888-4fd7-82b4-184d1b1c1569-crio-socket\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.214389 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.214292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5692a4-c888-4fd7-82b4-184d1b1c1569-data-volume\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.214389 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.214344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.234196 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.234137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" Jul 06 22:24:39.315840 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.315784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.315905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.315940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5d9\" (UniqueName: \"kubernetes.io/projected/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-api-access-bt5d9\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.315965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c5692a4-c888-4fd7-82b4-184d1b1c1569-crio-socket\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.316009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5692a4-c888-4fd7-82b4-184d1b1c1569-data-volume\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316551 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.316353 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:39.316551 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.316411 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c5692a4-c888-4fd7-82b4-184d1b1c1569-data-volume\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316551 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.316440 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:39.81641933 +0000 UTC m=+35.842833677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : configmap "kube-rbac-proxy" not found Jul 06 22:24:39.316551 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.316527 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:39.316551 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.316537 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c5692a4-c888-4fd7-82b4-184d1b1c1569-crio-socket\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.316881 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.316584 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:39.816565929 +0000 UTC m=+35.842980274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:39.325537 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.325436 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5d9\" (UniqueName: \"kubernetes.io/projected/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-api-access-bt5d9\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.517734 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.517513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:39.517933 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.517679 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:39.517933 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.517881 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls podName:e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e nodeName:}" failed. No retries permitted until 2026-07-06 22:24:40.517855789 +0000 UTC m=+36.544270132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls") pod "dns-default-tlk95" (UID: "e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e") : secret "dns-default-metrics-tls" not found Jul 06 22:24:39.820362 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.820314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.820569 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.820422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:39.820569 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.820470 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:39.820569 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.820505 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:39.820569 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.820544 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:40.820526238 +0000 UTC m=+36.846940580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:39.820569 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:39.820559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:40.820552504 +0000 UTC m=+36.846966846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : configmap "kube-rbac-proxy" not found Jul 06 22:24:39.934550 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.934517 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp"] Jul 06 22:24:39.937922 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:39.937896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ntp85"] Jul 06 22:24:40.033015 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:40.032888 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3264a6_1ef1_49d8_a9cf_3d479eb1466b.slice/crio-fbded1412f1b7e01a523d1e5edf60da51695e6ff2e15e2784ae7bfd0ca418501 WatchSource:0}: Error finding container fbded1412f1b7e01a523d1e5edf60da51695e6ff2e15e2784ae7bfd0ca418501: Status 404 returned error can't find the container with id fbded1412f1b7e01a523d1e5edf60da51695e6ff2e15e2784ae7bfd0ca418501 Jul 06 22:24:40.033516 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:40.033499 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70472f5_e8bd_4ddb_9ea4_0c4559febb51.slice/crio-63c1e385a0cd30b8cebf00f33b70eb7bc0c25ddf29e3fe1030023299cf0d0dd2 WatchSource:0}: Error finding container 63c1e385a0cd30b8cebf00f33b70eb7bc0c25ddf29e3fe1030023299cf0d0dd2: Status 404 returned error can't find the container with id 63c1e385a0cd30b8cebf00f33b70eb7bc0c25ddf29e3fe1030023299cf0d0dd2 Jul 06 22:24:40.516453 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.516416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:24:40.516892 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.516471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:24:40.516892 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.516416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:24:40.519131 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.519108 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:24:40.520253 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.520192 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zvxv7\"" Jul 06 22:24:40.520253 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.520212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:24:40.520434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.520277 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:24:40.520434 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.520313 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhlqr\"" Jul 06 22:24:40.520573 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.520553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:24:40.525512 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.525490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:40.525654 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.525628 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:40.525716 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.525683 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls podName:e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.525666522 +0000 UTC m=+38.552080865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls") pod "dns-default-tlk95" (UID: "e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e") : secret "dns-default-metrics-tls" not found Jul 06 22:24:40.679767 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.679722 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="a791655fef4120254b0049400f86e30125dfcb774b99af09c7286070f0fc7439" exitCode=0 Jul 06 22:24:40.679944 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.679804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"a791655fef4120254b0049400f86e30125dfcb774b99af09c7286070f0fc7439"} Jul 06 22:24:40.682194 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.682159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ntp85" event={"ID":"e70472f5-e8bd-4ddb-9ea4-0c4559febb51","Type":"ContainerStarted","Data":"63c1e385a0cd30b8cebf00f33b70eb7bc0c25ddf29e3fe1030023299cf0d0dd2"} Jul 06 22:24:40.683837 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.683814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" event={"ID":"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b","Type":"ContainerStarted","Data":"fbded1412f1b7e01a523d1e5edf60da51695e6ff2e15e2784ae7bfd0ca418501"} Jul 06 22:24:40.828920 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.828886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:40.829081 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:40.829054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:40.829140 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.829074 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:40.829204 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.829153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.829130713 +0000 UTC m=+38.855545070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:40.829204 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.829178 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:40.829307 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:40.829239 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.829222653 +0000 UTC m=+38.855637014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : configmap "kube-rbac-proxy" not found Jul 06 22:24:41.110914 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.110825 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5qrdf"] Jul 06 22:24:41.112747 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.112723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.116442 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.116411 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 22:24:41.116620 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.116572 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 22:24:41.116689 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.116671 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-r95xq\"" Jul 06 22:24:41.116886 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.116868 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 22:24:41.116992 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.116916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 22:24:41.130685 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.130618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5qrdf"] Jul 06 22:24:41.231878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.231835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/887c84fa-4325-4866-b988-680e94cf5a9d-signing-cabundle\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.231878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.231878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwh26\" (UniqueName: \"kubernetes.io/projected/887c84fa-4325-4866-b988-680e94cf5a9d-kube-api-access-fwh26\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.232148 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.232016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/887c84fa-4325-4866-b988-680e94cf5a9d-signing-key\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.332839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.332807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/887c84fa-4325-4866-b988-680e94cf5a9d-signing-cabundle\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.332839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.332842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwh26\" (UniqueName: \"kubernetes.io/projected/887c84fa-4325-4866-b988-680e94cf5a9d-kube-api-access-fwh26\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.333117 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.332893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/887c84fa-4325-4866-b988-680e94cf5a9d-signing-key\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.333655 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.333633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/887c84fa-4325-4866-b988-680e94cf5a9d-signing-cabundle\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.336952 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.336932 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/887c84fa-4325-4866-b988-680e94cf5a9d-signing-key\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.348712 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.348686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwh26\" (UniqueName: \"kubernetes.io/projected/887c84fa-4325-4866-b988-680e94cf5a9d-kube-api-access-fwh26\") pod \"service-ca-758f599c5-5qrdf\" (UID: \"887c84fa-4325-4866-b988-680e94cf5a9d\") " pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.423679 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.423628 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5qrdf" Jul 06 22:24:41.690109 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.690024 2576 generic.go:356] "Generic (PLEG): container finished" podID="89cdd929-92a4-450e-a67a-ebc2259983ec" containerID="9f41bac4932867aec7e27d08985ac220fa45bfe800813ac195c884d3ffaab965" exitCode=0 Jul 06 22:24:41.690109 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:41.690087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerDied","Data":"9f41bac4932867aec7e27d08985ac220fa45bfe800813ac195c884d3ffaab965"} Jul 06 22:24:42.030421 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.030169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xq8p6_c8479ad7-5197-44e5-b8a0-f6d111714378/dns-node-resolver/0.log" Jul 06 22:24:42.497249 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.497035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5qrdf"] Jul 06 22:24:42.500555 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:42.500520 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887c84fa_4325_4866_b988_680e94cf5a9d.slice/crio-d7182dcd89ccedcc700b1712b5346dde6be03df1687abc58d2423ab5dbb56986 WatchSource:0}: Error finding container d7182dcd89ccedcc700b1712b5346dde6be03df1687abc58d2423ab5dbb56986: Status 404 returned error can't find the container with id d7182dcd89ccedcc700b1712b5346dde6be03df1687abc58d2423ab5dbb56986 Jul 06 22:24:42.544651 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.544616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:42.544792 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.544729 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:42.544832 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.544793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls podName:e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e nodeName:}" failed. No retries permitted until 2026-07-06 22:24:46.544777849 +0000 UTC m=+42.571192192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls") pod "dns-default-tlk95" (UID: "e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e") : secret "dns-default-metrics-tls" not found Jul 06 22:24:42.694050 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.694015 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ntp85" event={"ID":"e70472f5-e8bd-4ddb-9ea4-0c4559febb51","Type":"ContainerStarted","Data":"d98875acd242d5d8915717edd1e91b1feb11baf779c69b0ab4e63cf42ca5867b"} Jul 06 22:24:42.695732 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.695702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" event={"ID":"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b","Type":"ContainerStarted","Data":"63e43aa078deab05ddaeebbe34a3302daa42f50f6630a107b7a53b872552bd27"} Jul 06 22:24:42.695845 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.695765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" event={"ID":"ad3264a6-1ef1-49d8-a9cf-3d479eb1466b","Type":"ContainerStarted","Data":"bc7dd7de5b721fc31b03712bf3c3c627fad9b781db7019e94610b4d6d4db97cb"} Jul 06 22:24:42.696801 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.696778 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5qrdf" event={"ID":"887c84fa-4325-4866-b988-680e94cf5a9d","Type":"ContainerStarted","Data":"d7182dcd89ccedcc700b1712b5346dde6be03df1687abc58d2423ab5dbb56986"} Jul 06 22:24:42.699940 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.699918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j29rx" event={"ID":"89cdd929-92a4-450e-a67a-ebc2259983ec","Type":"ContainerStarted","Data":"5bc9f91de7d3511d45cf862eb7baf7bb9c6d729e50bf8d6d0623b2d71d4f5729"} Jul 06 22:24:42.709352 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.709308 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ntp85" podStartSLOduration=2.414601624 podStartE2EDuration="4.709291992s" totalImagesPullingTime="2.294690368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:42.709021011 +0000 UTC m=+38.735435378" watchObservedRunningTime="2026-07-06 22:24:42.709291992 +0000 UTC m=+38.735706368" Jul 06 22:24:42.728958 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.728908 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j29rx" podStartSLOduration=8.185935258 podStartE2EDuration="38.728895382s" totalImagesPullingTime="30.542960124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:42.728354157 +0000 UTC m=+38.754768522" watchObservedRunningTime="2026-07-06 22:24:42.728895382 +0000 UTC m=+38.755309746" Jul 06 22:24:42.742569 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.742527 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bjqkp" podStartSLOduration=2.445974847 podStartE2EDuration="4.742513816s" totalImagesPullingTime="2.296538969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:42.741718782 +0000 UTC m=+38.768133148" watchObservedRunningTime="2026-07-06 22:24:42.742513816 +0000 UTC m=+38.768928180" Jul 06 22:24:42.847922 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.847880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:42.848114 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.847929 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:42.848114 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.848020 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:46.848003688 +0000 UTC m=+42.874418033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:42.848259 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:42.848178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:42.848324 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.848307 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:42.848400 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:42.848386 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:46.848365913 +0000 UTC m=+42.874780266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : configmap "kube-rbac-proxy" not found Jul 06 22:24:43.029482 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:43.029390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xhdwp_9a39f0e7-e0e5-4107-afcb-86938748c51b/node-ca/0.log" Jul 06 22:24:43.830404 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:43.830364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ntp85_e70472f5-e8bd-4ddb-9ea4-0c4559febb51/serve-healthcheck-canary/0.log" Jul 06 22:24:44.030311 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:44.030277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bjqkp_ad3264a6-1ef1-49d8-a9cf-3d479eb1466b/migrator/0.log" Jul 06 22:24:44.229682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:44.229651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bjqkp_ad3264a6-1ef1-49d8-a9cf-3d479eb1466b/graceful-termination/0.log" Jul 06 22:24:44.706905 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:44.706863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5qrdf" event={"ID":"887c84fa-4325-4866-b988-680e94cf5a9d","Type":"ContainerStarted","Data":"e3bb8ef5efa7a3bb1bccd8b6c865d810a6ff25a02c7a3ef431d1485aa432f0a9"} Jul 06 22:24:46.579429 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:46.579389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:46.579809 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:46.579526 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:46.579809 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:46.579593 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls podName:e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e nodeName:}" failed. No retries permitted until 2026-07-06 22:24:54.579578799 +0000 UTC m=+50.605993142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls") pod "dns-default-tlk95" (UID: "e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e") : secret "dns-default-metrics-tls" not found Jul 06 22:24:46.882092 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:46.882060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:46.882275 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:46.882149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:46.882275 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:46.882208 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:46.882275 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:24:46.882274 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls podName:8c5692a4-c888-4fd7-82b4-184d1b1c1569 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:54.882258963 +0000 UTC m=+50.908673305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7m8tk" (UID: "8c5692a4-c888-4fd7-82b4-184d1b1c1569") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:46.895001 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:46.894947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c5692a4-c888-4fd7-82b4-184d1b1c1569-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:54.634854 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.634811 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:54.638802 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.638776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e-metrics-tls\") pod \"dns-default-tlk95\" (UID: \"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e\") " pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:54.748645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.748604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlk95" Jul 06 22:24:54.882447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.882385 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5qrdf" podStartSLOduration=12.134729365 podStartE2EDuration="13.882358169s" totalImagesPullingTime="1.747628804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:44.722255954 +0000 UTC m=+40.748670342" watchObservedRunningTime="2026-07-06 22:24:54.882358169 +0000 UTC m=+50.908772534" Jul 06 22:24:54.882877 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.882858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlk95"] Jul 06 22:24:54.939018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.938956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:54.941267 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.941245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c5692a4-c888-4fd7-82b4-184d1b1c1569-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7m8tk\" (UID: \"8c5692a4-c888-4fd7-82b4-184d1b1c1569\") " pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:54.987587 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:54.987551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7m8tk" Jul 06 22:24:55.143253 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:55.143178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7m8tk"] Jul 06 22:24:55.146676 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:24:55.146647 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5692a4_c888_4fd7_82b4_184d1b1c1569.slice/crio-fad8e3be304602a0b324c825d9ceeb25ba66ff206769379ea71c093fb00c6a8c WatchSource:0}: Error finding container fad8e3be304602a0b324c825d9ceeb25ba66ff206769379ea71c093fb00c6a8c: Status 404 returned error can't find the container with id fad8e3be304602a0b324c825d9ceeb25ba66ff206769379ea71c093fb00c6a8c Jul 06 22:24:55.737428 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:55.737384 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m8tk" event={"ID":"8c5692a4-c888-4fd7-82b4-184d1b1c1569","Type":"ContainerStarted","Data":"fad8e3be304602a0b324c825d9ceeb25ba66ff206769379ea71c093fb00c6a8c"} Jul 06 22:24:55.738673 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:24:55.738640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlk95" event={"ID":"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e","Type":"ContainerStarted","Data":"8cc21de3b147b15d9381cef9a9fa8f3ffc848c1509d0a49196cc8f7efe530bb2"} Jul 06 22:25:00.663995 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.663941 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf"] Jul 06 22:25:00.665843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.665822 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq"] Jul 06 22:25:00.665972 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.665953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.667364 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.667345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.669187 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.669170 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 22:25:00.669187 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.669181 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 22:25:00.669320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.669302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 22:25:00.670061 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670041 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 22:25:00.670182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670132 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 22:25:00.670182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670141 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 22:25:00.670182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670160 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 22:25:00.670500 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670486 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 22:25:00.670621 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.670607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m5kwb\"" Jul 06 22:25:00.681934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.681908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq"] Jul 06 22:25:00.687224 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.687351 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.687351 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/761e30b7-308a-45bd-a086-67463e4938f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.687483 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.687528 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.687564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.687539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95x6h\" (UniqueName: \"kubernetes.io/projected/761e30b7-308a-45bd-a086-67463e4938f6-kube-api-access-95x6h\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.689115 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.689094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf"] Jul 06 22:25:00.762283 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.762244 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m8tk" event={"ID":"8c5692a4-c888-4fd7-82b4-184d1b1c1569","Type":"ContainerStarted","Data":"31ee867753d22c8d8a574fda9fc0de1ea7f2393d11b88ac5f4674c954ace2879"} Jul 06 22:25:00.762430 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.762314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m8tk" event={"ID":"8c5692a4-c888-4fd7-82b4-184d1b1c1569","Type":"ContainerStarted","Data":"03b703a6b71b65e9dc272b08f1c8f8dedb1661367772530b3ea2fc8e1c924be7"} Jul 06 22:25:00.763840 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.763810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlk95" event={"ID":"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e","Type":"ContainerStarted","Data":"d74e6f08f66f50c456fe95f2ab851060fc0c35afb6a40eedaef4b41925bae599"} Jul 06 22:25:00.763952 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.763855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlk95" event={"ID":"e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e","Type":"ContainerStarted","Data":"175aa21e9f635dc9ad5b2ffe1dd0c92995fa72756997d74187b784f43ae6288e"} Jul 06 22:25:00.764056 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.763955 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tlk95" Jul 06 22:25:00.788449 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c33ab245-56c4-4e12-becb-83a24a0e599d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pc7\" (UniqueName: \"kubernetes.io/projected/c33ab245-56c4-4e12-becb-83a24a0e599d-kube-api-access-h9pc7\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788557 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95x6h\" (UniqueName: \"kubernetes.io/projected/761e30b7-308a-45bd-a086-67463e4938f6-kube-api-access-95x6h\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.788649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.788956 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.788669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/761e30b7-308a-45bd-a086-67463e4938f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.790115 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.790085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/761e30b7-308a-45bd-a086-67463e4938f6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.790300 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.790163 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tlk95" podStartSLOduration=18.021962357 podStartE2EDuration="22.790145992s" totalImagesPullingTime="4.768183635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:00.789935399 +0000 UTC m=+56.816349764" watchObservedRunningTime="2026-07-06 22:25:00.790145992 +0000 UTC m=+56.816560338" Jul 06 22:25:00.791622 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.791593 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-ca\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.791714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.791666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.791714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.791682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.791785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.791718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/761e30b7-308a-45bd-a086-67463e4938f6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.800580 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.800555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95x6h\" (UniqueName: \"kubernetes.io/projected/761e30b7-308a-45bd-a086-67463e4938f6-kube-api-access-95x6h\") pod \"cluster-proxy-proxy-agent-59b8bf89c4-j8tnf\" (UID: \"761e30b7-308a-45bd-a086-67463e4938f6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.890034 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.889962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c33ab245-56c4-4e12-becb-83a24a0e599d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.890034 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.890042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pc7\" (UniqueName: \"kubernetes.io/projected/c33ab245-56c4-4e12-becb-83a24a0e599d-kube-api-access-h9pc7\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.892511 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.892475 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c33ab245-56c4-4e12-becb-83a24a0e599d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.898735 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.898699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pc7\" (UniqueName: \"kubernetes.io/projected/c33ab245-56c4-4e12-becb-83a24a0e599d-kube-api-access-h9pc7\") pod \"managed-serviceaccount-addon-agent-76977b88f8-qbslq\" (UID: \"c33ab245-56c4-4e12-becb-83a24a0e599d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:00.985170 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.985074 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" Jul 06 22:25:00.990636 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:00.990616 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" Jul 06 22:25:01.126144 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:01.126103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf"] Jul 06 22:25:01.129699 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:01.129674 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod761e30b7_308a_45bd_a086_67463e4938f6.slice/crio-58f87b5c3e46ed4a899180d58346e6c4490557caf9ab861e29c2ffd475fbf855 WatchSource:0}: Error finding container 58f87b5c3e46ed4a899180d58346e6c4490557caf9ab861e29c2ffd475fbf855: Status 404 returned error can't find the container with id 58f87b5c3e46ed4a899180d58346e6c4490557caf9ab861e29c2ffd475fbf855 Jul 06 22:25:01.138891 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:01.138801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq"] Jul 06 22:25:01.143178 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:01.143148 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc33ab245_56c4_4e12_becb_83a24a0e599d.slice/crio-aafc3d98a0ea5a9efdc47244941a439f0c3e87b0ce2233d3bd69e92c6fc6efc5 WatchSource:0}: Error finding container aafc3d98a0ea5a9efdc47244941a439f0c3e87b0ce2233d3bd69e92c6fc6efc5: Status 404 returned error can't find the container with id aafc3d98a0ea5a9efdc47244941a439f0c3e87b0ce2233d3bd69e92c6fc6efc5 Jul 06 22:25:01.768160 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:01.768117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" event={"ID":"c33ab245-56c4-4e12-becb-83a24a0e599d","Type":"ContainerStarted","Data":"aafc3d98a0ea5a9efdc47244941a439f0c3e87b0ce2233d3bd69e92c6fc6efc5"} Jul 06 22:25:01.769339 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:01.769298 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" event={"ID":"761e30b7-308a-45bd-a086-67463e4938f6","Type":"ContainerStarted","Data":"58f87b5c3e46ed4a899180d58346e6c4490557caf9ab861e29c2ffd475fbf855"} Jul 06 22:25:04.677052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:04.677018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m8grd" Jul 06 22:25:06.791242 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.791210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7m8tk" event={"ID":"8c5692a4-c888-4fd7-82b4-184d1b1c1569","Type":"ContainerStarted","Data":"8a3587d8528da5a804ff0e1f13b5dc3555446bc2395828142e1512d0abb6dd5c"} Jul 06 22:25:06.792634 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.792614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" event={"ID":"c33ab245-56c4-4e12-becb-83a24a0e599d","Type":"ContainerStarted","Data":"6b064096cbc537513ce893aafbab5b71ba1434e0931f59c676451a04f6c339ad"} Jul 06 22:25:06.794660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.794632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" event={"ID":"761e30b7-308a-45bd-a086-67463e4938f6","Type":"ContainerStarted","Data":"1d7ea4e042fffb468fbdc4f991f544859e66526387f67a8f9a3eb4d91b7496ab"} Jul 06 22:25:06.794773 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.794671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" event={"ID":"761e30b7-308a-45bd-a086-67463e4938f6","Type":"ContainerStarted","Data":"98c700a52fa9abfe8facadb25affb4ab8723528f263e25c3300eab8afab000cc"} Jul 06 22:25:06.794773 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.794691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" event={"ID":"761e30b7-308a-45bd-a086-67463e4938f6","Type":"ContainerStarted","Data":"9014c8a7a27b8908a70e36b5ad4cb3ff956119855bc49b13e95d8b55f9a80d77"} Jul 06 22:25:06.808953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.808910 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7m8tk" podStartSLOduration=16.793250742 podStartE2EDuration="27.808895284s" totalImagesPullingTime="11.015644542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:06.808090002 +0000 UTC m=+62.834504366" watchObservedRunningTime="2026-07-06 22:25:06.808895284 +0000 UTC m=+62.835309647" Jul 06 22:25:06.825042 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.824965 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b8bf89c4-j8tnf" podStartSLOduration=1.652319901 podStartE2EDuration="6.824950601s" totalImagesPullingTime="5.1726307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:06.824124703 +0000 UTC m=+62.850539080" watchObservedRunningTime="2026-07-06 22:25:06.824950601 +0000 UTC m=+62.851364964" Jul 06 22:25:06.840118 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:06.840072 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76977b88f8-qbslq" podStartSLOduration=1.690330391 podStartE2EDuration="6.840056213s" totalImagesPullingTime="5.149725822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:06.838911911 +0000 UTC m=+62.865326274" watchObservedRunningTime="2026-07-06 22:25:06.840056213 +0000 UTC m=+62.866470576" Jul 06 22:25:10.168346 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.168287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:25:10.170966 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.170948 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:25:10.181162 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.181139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd-metrics-certs\") pod \"network-metrics-daemon-v2h59\" (UID: \"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd\") " pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:25:10.238720 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.238694 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zhlqr\"" Jul 06 22:25:10.246888 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.246861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2h59" Jul 06 22:25:10.268953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.268924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:25:10.269132 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.268970 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:25:10.271761 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.271743 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:25:10.271932 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.271916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:25:10.281287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.281259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b3f4825-2adc-4709-b51c-1cdb45711326-original-pull-secret\") pod \"global-pull-secret-syncer-plfpg\" (UID: \"9b3f4825-2adc-4709-b51c-1cdb45711326\") " pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:25:10.281599 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.281578 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:25:10.291972 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.291941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplk4\" (UniqueName: \"kubernetes.io/projected/c9fe3494-24a7-4d74-8ffd-13b52d788547-kube-api-access-jplk4\") pod \"network-check-target-z7wf2\" (UID: \"c9fe3494-24a7-4d74-8ffd-13b52d788547\") " pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:25:10.373778 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.373749 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2h59"] Jul 06 22:25:10.377067 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:10.377038 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d5b891_7c15_4ecd_a6f9_8b7f0e941cbd.slice/crio-cf3497882bb9758bec7975fc0d3dd221fed895e5d95537cb7c0dd28d71ef19f8 WatchSource:0}: Error finding container cf3497882bb9758bec7975fc0d3dd221fed895e5d95537cb7c0dd28d71ef19f8: Status 404 returned error can't find the container with id cf3497882bb9758bec7975fc0d3dd221fed895e5d95537cb7c0dd28d71ef19f8 Jul 06 22:25:10.533054 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.532959 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zvxv7\"" Jul 06 22:25:10.541064 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.541048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:25:10.543389 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.543372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-plfpg" Jul 06 22:25:10.669996 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.669947 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7wf2"] Jul 06 22:25:10.673543 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:10.673517 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fe3494_24a7_4d74_8ffd_13b52d788547.slice/crio-47c08694e10b673a65162aeede3e302a74c33e76d45cae0baf8badfb84061afb WatchSource:0}: Error finding container 47c08694e10b673a65162aeede3e302a74c33e76d45cae0baf8badfb84061afb: Status 404 returned error can't find the container with id 47c08694e10b673a65162aeede3e302a74c33e76d45cae0baf8badfb84061afb Jul 06 22:25:10.684869 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.684846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-plfpg"] Jul 06 22:25:10.687307 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:10.687286 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3f4825_2adc_4709_b51c_1cdb45711326.slice/crio-0f4793bd078028e7958989350aaa3db16dd7b3c2fd0bdf0c99efd26e09b527ab WatchSource:0}: Error finding container 0f4793bd078028e7958989350aaa3db16dd7b3c2fd0bdf0c99efd26e09b527ab: Status 404 returned error can't find the container with id 0f4793bd078028e7958989350aaa3db16dd7b3c2fd0bdf0c99efd26e09b527ab Jul 06 22:25:10.771247 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.771216 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tlk95" Jul 06 22:25:10.808818 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.808737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7wf2" event={"ID":"c9fe3494-24a7-4d74-8ffd-13b52d788547","Type":"ContainerStarted","Data":"47c08694e10b673a65162aeede3e302a74c33e76d45cae0baf8badfb84061afb"} Jul 06 22:25:10.810030 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.810004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plfpg" event={"ID":"9b3f4825-2adc-4709-b51c-1cdb45711326","Type":"ContainerStarted","Data":"0f4793bd078028e7958989350aaa3db16dd7b3c2fd0bdf0c99efd26e09b527ab"} Jul 06 22:25:10.811153 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:10.811129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2h59" event={"ID":"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd","Type":"ContainerStarted","Data":"cf3497882bb9758bec7975fc0d3dd221fed895e5d95537cb7c0dd28d71ef19f8"} Jul 06 22:25:11.817925 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:11.817887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2h59" event={"ID":"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd","Type":"ContainerStarted","Data":"daab578c4f99a80e23c23b5a17d834082c4a474361a58d26d4a1365cbe8a3d34"} Jul 06 22:25:12.824458 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:12.824414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2h59" event={"ID":"f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd","Type":"ContainerStarted","Data":"f889c9e9f5aec761dcbab7eb3de943a2591c581695a230fba16de7d4c9039eb2"} Jul 06 22:25:12.841295 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:12.841235 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v2h59" podStartSLOduration=67.589135007 podStartE2EDuration="1m8.841212209s" totalImagesPullingTime="1.252077202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:12.840213708 +0000 UTC m=+68.866628067" watchObservedRunningTime="2026-07-06 22:25:12.841212209 +0000 UTC m=+68.867626574" Jul 06 22:25:15.910143 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.910114 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f5q64"] Jul 06 22:25:15.912001 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.911965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:15.914403 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.914383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:25:15.915953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.915935 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:25:15.915953 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.915946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:25:15.916760 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.916737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wsr9r\"" Jul 06 22:25:15.916848 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.916776 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:25:15.916848 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.916833 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:25:15.917393 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:15.917375 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:25:16.010815 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-metrics-client-ca\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.010918 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-wtmp\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.010918 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-sys\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.010918 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-root\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.011041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-textfile\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.011041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.011041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.010996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-tls\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.011140 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.011122 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.011180 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.011153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdk2q\" (UniqueName: \"kubernetes.io/projected/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-kube-api-access-mdk2q\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.111936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-tls\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdk2q\" (UniqueName: \"kubernetes.io/projected/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-kube-api-access-mdk2q\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-metrics-client-ca\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-wtmp\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-sys\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-root\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-textfile\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-sys\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-root\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.112950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-wtmp\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.113144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113942 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.113417 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-textfile\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.113942 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.113449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-metrics-client-ca\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.115638 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.115612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.115772 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.115666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-node-exporter-tls\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.121632 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.121607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdk2q\" (UniqueName: \"kubernetes.io/projected/7af163fd-380c-40c1-8bf5-09ebf2b7ac36-kube-api-access-mdk2q\") pod \"node-exporter-f5q64\" (UID: \"7af163fd-380c-40c1-8bf5-09ebf2b7ac36\") " pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.222146 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.222050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5q64" Jul 06 22:25:16.231927 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:25:16.231898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af163fd_380c_40c1_8bf5_09ebf2b7ac36.slice/crio-1140c482616ca801b62d96c13afcd8689241ffb04d22fe6bcf927c3ec4a2bc3e WatchSource:0}: Error finding container 1140c482616ca801b62d96c13afcd8689241ffb04d22fe6bcf927c3ec4a2bc3e: Status 404 returned error can't find the container with id 1140c482616ca801b62d96c13afcd8689241ffb04d22fe6bcf927c3ec4a2bc3e Jul 06 22:25:16.843146 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.842855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7wf2" event={"ID":"c9fe3494-24a7-4d74-8ffd-13b52d788547","Type":"ContainerStarted","Data":"ea3dd83d9b5bcc2bd97bc4c3cfcfb19e79dbddeb210d805e6eaf5cdfeea21d52"} Jul 06 22:25:16.843146 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.843098 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:25:16.844439 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.844409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-plfpg" event={"ID":"9b3f4825-2adc-4709-b51c-1cdb45711326","Type":"ContainerStarted","Data":"0d0b7fcce52f583471a005b801d9ba63c64be6dc41069bc9efd467f66b5ec2f9"} Jul 06 22:25:16.845774 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.845723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5q64" event={"ID":"7af163fd-380c-40c1-8bf5-09ebf2b7ac36","Type":"ContainerStarted","Data":"1140c482616ca801b62d96c13afcd8689241ffb04d22fe6bcf927c3ec4a2bc3e"} Jul 06 22:25:16.935723 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.935661 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7wf2" podStartSLOduration=67.674394337 podStartE2EDuration="1m12.935641121s" totalImagesPullingTime="5.261246784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:16.884727676 +0000 UTC m=+72.911142041" watchObservedRunningTime="2026-07-06 22:25:16.935641121 +0000 UTC m=+72.962055486" Jul 06 22:25:16.936198 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:16.935831 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-plfpg" podStartSLOduration=67.681389034 podStartE2EDuration="1m12.935824721s" totalImagesPullingTime="5.254435687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:16.935415275 +0000 UTC m=+72.961829641" watchObservedRunningTime="2026-07-06 22:25:16.935824721 +0000 UTC m=+72.962239086" Jul 06 22:25:17.850893 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:17.850855 2576 generic.go:356] "Generic (PLEG): container finished" podID="7af163fd-380c-40c1-8bf5-09ebf2b7ac36" containerID="9b7d4738c521e4c083cebf97e2373766d64fb73eed4054479095307fa2e37607" exitCode=0 Jul 06 22:25:17.851085 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:17.850949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5q64" event={"ID":"7af163fd-380c-40c1-8bf5-09ebf2b7ac36","Type":"ContainerDied","Data":"9b7d4738c521e4c083cebf97e2373766d64fb73eed4054479095307fa2e37607"} Jul 06 22:25:18.856046 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:18.856013 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5q64" event={"ID":"7af163fd-380c-40c1-8bf5-09ebf2b7ac36","Type":"ContainerStarted","Data":"5584abe12e6ea9328aefda93c2c259e05d3f6550ff8b873926b6bc7d51fd313f"} Jul 06 22:25:18.856527 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:18.856072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5q64" event={"ID":"7af163fd-380c-40c1-8bf5-09ebf2b7ac36","Type":"ContainerStarted","Data":"951d164fc777bd4e0c98f37aeef8d3156054c23662f80962de0c3727b9af5507"} Jul 06 22:25:18.875901 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:18.875850 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f5q64" podStartSLOduration=2.712899662 podStartE2EDuration="3.875833244s" totalImagesPullingTime="1.099598445s" totalInitContainerRuntime="63.335137ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:18.875167715 +0000 UTC m=+74.901582079" watchObservedRunningTime="2026-07-06 22:25:18.875833244 +0000 UTC m=+74.902247608" Jul 06 22:25:47.853905 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:25:47.853877 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7wf2" Jul 06 22:29:04.365943 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:04.365914 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:29:17.359082 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.359042 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lqqgx"] Jul 06 22:29:17.360850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.360836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.364327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.364305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 22:29:17.365382 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.365365 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 22:29:17.365494 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.365426 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vlt9x\"" Jul 06 22:29:17.401166 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.401123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qr8\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-kube-api-access-f4qr8\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.401166 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.401172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.406931 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.406896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lqqgx"] Jul 06 22:29:17.502464 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.502427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.502637 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.502506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qr8\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-kube-api-access-f4qr8\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.515318 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.515288 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.517744 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.517722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qr8\" (UniqueName: \"kubernetes.io/projected/f1711f03-978d-4c1f-a18c-2ebe28b4ff8f-kube-api-access-f4qr8\") pod \"cert-manager-webhook-6486ffd997-lqqgx\" (UID: \"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.669291 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.669253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:17.803866 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.803834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lqqgx"] Jul 06 22:29:17.807108 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:29:17.807082 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1711f03_978d_4c1f_a18c_2ebe28b4ff8f.slice/crio-fa4f8353f2affe493e20c879014a154172db5d9c1d911b6067b6cafd7fdd07f7 WatchSource:0}: Error finding container fa4f8353f2affe493e20c879014a154172db5d9c1d911b6067b6cafd7fdd07f7: Status 404 returned error can't find the container with id fa4f8353f2affe493e20c879014a154172db5d9c1d911b6067b6cafd7fdd07f7 Jul 06 22:29:17.809060 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:17.809041 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:29:18.630734 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:18.630675 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" event={"ID":"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f","Type":"ContainerStarted","Data":"fa4f8353f2affe493e20c879014a154172db5d9c1d911b6067b6cafd7fdd07f7"} Jul 06 22:29:20.642713 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:20.642680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" event={"ID":"f1711f03-978d-4c1f-a18c-2ebe28b4ff8f","Type":"ContainerStarted","Data":"b791f8dd25802bfa11d11e1e61854e8e571bddda94bde8c2b7cd8aaad58a929e"} Jul 06 22:29:20.643111 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:20.642788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:20.707763 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:20.707710 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" podStartSLOduration=0.94512814 podStartE2EDuration="3.707696255s" totalImagesPullingTime="2.762568115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:20.707330094 +0000 UTC m=+316.733744457" watchObservedRunningTime="2026-07-06 22:29:20.707696255 +0000 UTC m=+316.734110619" Jul 06 22:29:26.648379 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:26.648348 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lqqgx" Jul 06 22:29:42.456476 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.456437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4"] Jul 06 22:29:42.460877 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.460847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.464124 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.464089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:29:42.464365 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.464349 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:29:42.464595 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.464580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:29:42.464813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.464789 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nd4qr\"" Jul 06 22:29:42.464909 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.464818 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:29:42.477772 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.477730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4"] Jul 06 22:29:42.578907 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.578868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.579102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.578914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-manifests\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.579102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.579013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.579102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.579060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbnng\" (UniqueName: \"kubernetes.io/projected/7ed5ae59-844f-4123-904a-86f9802ef890-kube-api-access-jbnng\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.579102 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.579092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-tmp\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680101 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680101 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbnng\" (UniqueName: \"kubernetes.io/projected/7ed5ae59-844f-4123-904a-86f9802ef890-kube-api-access-jbnng\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680378 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-tmp\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680378 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680378 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-manifests\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.680711 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.680681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-manifests\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.682647 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.682613 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ed5ae59-844f-4123-904a-86f9802ef890-tmp\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.682847 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.682827 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.682888 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.682857 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed5ae59-844f-4123-904a-86f9802ef890-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.689163 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.689134 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbnng\" (UniqueName: \"kubernetes.io/projected/7ed5ae59-844f-4123-904a-86f9802ef890-kube-api-access-jbnng\") pod \"opendatahub-operator-controller-manager-85fcf54485-vrls4\" (UID: \"7ed5ae59-844f-4123-904a-86f9802ef890\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.771363 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.771320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:42.941673 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:42.941636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4"] Jul 06 22:29:42.944858 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:29:42.944829 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed5ae59_844f_4123_904a_86f9802ef890.slice/crio-e6370ce75432a98fd510906c8412723ae0caf82b1c7150de325eabe942866d61 WatchSource:0}: Error finding container e6370ce75432a98fd510906c8412723ae0caf82b1c7150de325eabe942866d61: Status 404 returned error can't find the container with id e6370ce75432a98fd510906c8412723ae0caf82b1c7150de325eabe942866d61 Jul 06 22:29:43.721842 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:43.721728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" event={"ID":"7ed5ae59-844f-4123-904a-86f9802ef890","Type":"ContainerStarted","Data":"e6370ce75432a98fd510906c8412723ae0caf82b1c7150de325eabe942866d61"} Jul 06 22:29:46.733844 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:46.733805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" event={"ID":"7ed5ae59-844f-4123-904a-86f9802ef890","Type":"ContainerStarted","Data":"d1acb0ee07bdf1f153d8a16249a223b66df82d07c7a72444b7e099964518d54e"} Jul 06 22:29:47.738859 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:47.738817 2576 generic.go:356] "Generic (PLEG): container finished" podID="7ed5ae59-844f-4123-904a-86f9802ef890" containerID="d1acb0ee07bdf1f153d8a16249a223b66df82d07c7a72444b7e099964518d54e" exitCode=0 Jul 06 22:29:47.739310 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:47.738901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" event={"ID":"7ed5ae59-844f-4123-904a-86f9802ef890","Type":"ContainerDied","Data":"d1acb0ee07bdf1f153d8a16249a223b66df82d07c7a72444b7e099964518d54e"} Jul 06 22:29:48.745343 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:48.745303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" event={"ID":"7ed5ae59-844f-4123-904a-86f9802ef890","Type":"ContainerStarted","Data":"8ad8df382ad8438fea9f7aa2dd5c1a1dce23ab931ebcf40e0c7afceb4a9f3ec5"} Jul 06 22:29:48.745814 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:48.745440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:29:48.772353 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:48.772301 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" podStartSLOduration=2.387938679 podStartE2EDuration="6.77228314s" totalImagesPullingTime="3.285881467s" totalInitContainerRuntime="1.098462994s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:48.770110769 +0000 UTC m=+344.796525133" watchObservedRunningTime="2026-07-06 22:29:48.77228314 +0000 UTC m=+344.798697504" Jul 06 22:29:59.751935 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:29:59.751907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcf54485-vrls4" Jul 06 22:30:02.341258 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.341219 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jl2v8"] Jul 06 22:30:02.344571 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.344545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:02.347222 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.347200 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 22:30:02.347348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.347268 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-92qmm\"" Jul 06 22:30:02.353166 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.353136 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jl2v8"] Jul 06 22:30:02.424469 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.424429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:02.424469 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.424470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755j5\" (UniqueName: \"kubernetes.io/projected/0921b309-f093-4b4e-a8a7-0cdf615c0476-kube-api-access-755j5\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:02.525674 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.525634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:02.525674 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.525678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755j5\" (UniqueName: \"kubernetes.io/projected/0921b309-f093-4b4e-a8a7-0cdf615c0476-kube-api-access-755j5\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:02.525945 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:30:02.525787 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 22:30:02.525945 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:30:02.525852 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert podName:0921b309-f093-4b4e-a8a7-0cdf615c0476 nodeName:}" failed. No retries permitted until 2026-07-06 22:30:03.025833151 +0000 UTC m=+359.052247496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert") pod "odh-model-controller-7998c67dc4-jl2v8" (UID: "0921b309-f093-4b4e-a8a7-0cdf615c0476") : secret "odh-model-controller-webhook-cert" not found Jul 06 22:30:02.536217 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:02.536175 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755j5\" (UniqueName: \"kubernetes.io/projected/0921b309-f093-4b4e-a8a7-0cdf615c0476-kube-api-access-755j5\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:03.029619 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.029588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:03.032189 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.032150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0921b309-f093-4b4e-a8a7-0cdf615c0476-cert\") pod \"odh-model-controller-7998c67dc4-jl2v8\" (UID: \"0921b309-f093-4b4e-a8a7-0cdf615c0476\") " pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:03.212755 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.212712 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp"] Jul 06 22:30:03.216399 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.216369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.219239 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.219185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:30:03.219239 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.219185 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 22:30:03.219732 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.219700 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zd6q2\"" Jul 06 22:30:03.220224 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.220207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:30:03.220300 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.220222 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:30:03.245827 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.245794 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp"] Jul 06 22:30:03.256017 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.255959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:03.331811 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.331778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tmp\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.331967 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.331828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tls-certs\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.331967 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.331921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk95b\" (UniqueName: \"kubernetes.io/projected/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-kube-api-access-sk95b\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.396644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.396114 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jl2v8"] Jul 06 22:30:03.402594 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:30:03.402560 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0921b309_f093_4b4e_a8a7_0cdf615c0476.slice/crio-a32f61bbfa3460a40624500253c1ba80397414388cfd1597c1221d4080874d32 WatchSource:0}: Error finding container a32f61bbfa3460a40624500253c1ba80397414388cfd1597c1221d4080874d32: Status 404 returned error can't find the container with id a32f61bbfa3460a40624500253c1ba80397414388cfd1597c1221d4080874d32 Jul 06 22:30:03.433060 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.433014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tls-certs\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.433250 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.433071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk95b\" (UniqueName: \"kubernetes.io/projected/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-kube-api-access-sk95b\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.433250 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.433130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tmp\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.435417 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.435387 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tmp\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.435537 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.435512 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-tls-certs\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.440605 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.440577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk95b\" (UniqueName: \"kubernetes.io/projected/b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3-kube-api-access-sk95b\") pod \"kube-auth-proxy-7f6975f4cc-vljvp\" (UID: \"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3\") " pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.526907 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.526859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" Jul 06 22:30:03.658370 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.658326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp"] Jul 06 22:30:03.660854 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:30:03.660825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ab1f96_8212_4f0e_afeb_6c20f0ef73f3.slice/crio-283fc3fb8f54aba2bb597aa501b80f834867c442ab7c662018da41a72c22ea37 WatchSource:0}: Error finding container 283fc3fb8f54aba2bb597aa501b80f834867c442ab7c662018da41a72c22ea37: Status 404 returned error can't find the container with id 283fc3fb8f54aba2bb597aa501b80f834867c442ab7c662018da41a72c22ea37 Jul 06 22:30:03.797888 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.797853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" event={"ID":"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3","Type":"ContainerStarted","Data":"283fc3fb8f54aba2bb597aa501b80f834867c442ab7c662018da41a72c22ea37"} Jul 06 22:30:03.798850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:03.798829 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" event={"ID":"0921b309-f093-4b4e-a8a7-0cdf615c0476","Type":"ContainerStarted","Data":"a32f61bbfa3460a40624500253c1ba80397414388cfd1597c1221d4080874d32"} Jul 06 22:30:06.701496 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.701472 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:30:06.816788 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.816737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" event={"ID":"0921b309-f093-4b4e-a8a7-0cdf615c0476","Type":"ContainerStarted","Data":"d9e0ad05d43d0cc74e4289bd1b5414856e5eb4a12b56a9f8a90541324965502a"} Jul 06 22:30:06.817027 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.816843 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:06.818395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.818365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" event={"ID":"b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3","Type":"ContainerStarted","Data":"52073ea0194110e05267825c54ec9f50642c0f341b5c8ed41df3bd189fec9d80"} Jul 06 22:30:06.832601 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.832547 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" podStartSLOduration=1.537600126 podStartE2EDuration="4.832529302s" totalImagesPullingTime="3.294929176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:06.830738394 +0000 UTC m=+362.857152770" watchObservedRunningTime="2026-07-06 22:30:06.832529302 +0000 UTC m=+362.858943666" Jul 06 22:30:06.845259 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:06.845202 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f6975f4cc-vljvp" podStartSLOduration=0.809301724 podStartE2EDuration="3.845184178s" totalImagesPullingTime="3.035882454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:06.844198204 +0000 UTC m=+362.870612583" watchObservedRunningTime="2026-07-06 22:30:06.845184178 +0000 UTC m=+362.871598541" Jul 06 22:30:09.056924 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.056883 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd"] Jul 06 22:30:09.060370 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.060348 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.062938 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.062912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8q9hg\"" Jul 06 22:30:09.063104 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.062958 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 22:30:09.073353 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.073328 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd"] Jul 06 22:30:09.183479 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183479 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5524t\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-kube-api-access-5524t\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183673 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.183750 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183734 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.184036 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.183779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285134 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5524t\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-kube-api-access-5524t\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.285793 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.286193 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.285912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.286193 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.286004 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.286193 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.286016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.286400 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.286301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.288112 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.288094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.288359 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.288337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.297841 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.297793 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5524t\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-kube-api-access-5524t\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.298641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.298618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d747c0e-f59e-4b18-84d6-26dafab9d7e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678tffjd\" (UID: \"1d747c0e-f59e-4b18-84d6-26dafab9d7e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.371774 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.371678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:09.534423 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.534389 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd"] Jul 06 22:30:09.538887 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:30:09.538856 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d747c0e_f59e_4b18_84d6_26dafab9d7e5.slice/crio-4fa9326de89ec8aba1cc9ba274aeea36a8f3420b2a71dafc79ea67d8e084b00d WatchSource:0}: Error finding container 4fa9326de89ec8aba1cc9ba274aeea36a8f3420b2a71dafc79ea67d8e084b00d: Status 404 returned error can't find the container with id 4fa9326de89ec8aba1cc9ba274aeea36a8f3420b2a71dafc79ea67d8e084b00d Jul 06 22:30:09.833105 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:09.833066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" event={"ID":"1d747c0e-f59e-4b18-84d6-26dafab9d7e5","Type":"ContainerStarted","Data":"4fa9326de89ec8aba1cc9ba274aeea36a8f3420b2a71dafc79ea67d8e084b00d"} Jul 06 22:30:11.156597 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.156560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv"] Jul 06 22:30:11.163839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.163807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.168169 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168132 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qjwhf\"" Jul 06 22:30:11.168169 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168152 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 22:30:11.168401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168184 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:30:11.168401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168132 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 22:30:11.168401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168132 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 22:30:11.168401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.168139 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 22:30:11.175040 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.175013 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv"] Jul 06 22:30:11.302556 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.302516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-manager-config\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.302556 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.302557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.302809 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.302607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxnf\" (UniqueName: \"kubernetes.io/projected/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-kube-api-access-8nxnf\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.302809 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.302631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-metrics-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.404046 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.404011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxnf\" (UniqueName: \"kubernetes.io/projected/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-kube-api-access-8nxnf\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.404219 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.404051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-metrics-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.404219 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.404129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-manager-config\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.404219 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.404154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.404951 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.404923 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-manager-config\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.407164 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.407099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.407294 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.407246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-metrics-cert\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.414247 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.414203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxnf\" (UniqueName: \"kubernetes.io/projected/cd1aeaea-f6f1-457c-b337-b5b64288f8ee-kube-api-access-8nxnf\") pod \"lws-controller-manager-67b7b4db78-jg9lv\" (UID: \"cd1aeaea-f6f1-457c-b337-b5b64288f8ee\") " pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.476223 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.476187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:11.895676 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.895636 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:30:11.895782 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.895714 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:30:11.895782 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.895746 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:30:11.966720 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:11.966687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv"] Jul 06 22:30:11.969962 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:30:11.969933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1aeaea_f6f1_457c_b337_b5b64288f8ee.slice/crio-c3ced730cb072b8e9e1368ed85028c86ded26a6f371f8204cc983c8f58238fd5 WatchSource:0}: Error finding container c3ced730cb072b8e9e1368ed85028c86ded26a6f371f8204cc983c8f58238fd5: Status 404 returned error can't find the container with id c3ced730cb072b8e9e1368ed85028c86ded26a6f371f8204cc983c8f58238fd5 Jul 06 22:30:12.846362 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:12.846320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" event={"ID":"1d747c0e-f59e-4b18-84d6-26dafab9d7e5","Type":"ContainerStarted","Data":"6f1d845830f975282539c08a70ed9b81b1d9f8c9602b01e44454e6c573b97932"} Jul 06 22:30:12.847574 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:12.847547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" event={"ID":"cd1aeaea-f6f1-457c-b337-b5b64288f8ee","Type":"ContainerStarted","Data":"c3ced730cb072b8e9e1368ed85028c86ded26a6f371f8204cc983c8f58238fd5"} Jul 06 22:30:12.953503 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:12.953444 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" podStartSLOduration=1.599156882 podStartE2EDuration="3.953426903s" totalImagesPullingTime="2.354270021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:12.952099605 +0000 UTC m=+368.978513969" watchObservedRunningTime="2026-07-06 22:30:12.953426903 +0000 UTC m=+368.979841266" Jul 06 22:30:13.372070 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:13.372028 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:13.376843 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:13.376810 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:13.851477 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:13.851390 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:13.852394 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:13.852376 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678tffjd" Jul 06 22:30:17.824501 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:17.824467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jl2v8" Jul 06 22:30:21.883422 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:21.883387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" event={"ID":"cd1aeaea-f6f1-457c-b337-b5b64288f8ee","Type":"ContainerStarted","Data":"bbc5e94aaf9997a1a8fbfa4b6fe32807b032c8b5ce7fb56a79a075061324b87d"} Jul 06 22:30:21.883865 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:21.883451 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:30:21.900905 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:21.900848 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" podStartSLOduration=1.7770013439999999 podStartE2EDuration="10.900828177s" totalImagesPullingTime="9.123826833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:21.899739666 +0000 UTC m=+377.926154034" watchObservedRunningTime="2026-07-06 22:30:21.900828177 +0000 UTC m=+377.927242544" Jul 06 22:30:32.889679 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:30:32.889646 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67b7b4db78-jg9lv" Jul 06 22:32:50.738307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.738267 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:50.741673 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.741643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:50.744466 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.744439 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:32:50.744628 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.744438 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gbx8m\"" Jul 06 22:32:50.744628 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.744443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:32:50.749553 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.749369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:50.938752 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.938715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5qp\" (UniqueName: \"kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp\") pod \"authorino-58cd955675-qbdpf\" (UID: \"9d1c3c3f-389a-4cd0-8897-e870cea0e059\") " pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:50.952142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.952099 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:32:50.955136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.955113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:32:50.965817 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:50.965788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:32:51.039358 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.039263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5qp\" (UniqueName: \"kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp\") pod \"authorino-58cd955675-qbdpf\" (UID: \"9d1c3c3f-389a-4cd0-8897-e870cea0e059\") " pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:51.049152 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.049124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5qp\" (UniqueName: \"kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp\") pod \"authorino-58cd955675-qbdpf\" (UID: \"9d1c3c3f-389a-4cd0-8897-e870cea0e059\") " pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:51.052844 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.052806 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:51.140534 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.140293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g258p\" (UniqueName: \"kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p\") pod \"authorino-75d556bd77-5x4hp\" (UID: \"8f406ece-0409-4964-b2d2-1c097ae4c161\") " pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:32:51.198338 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.198116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:51.201256 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:32:51.201227 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d1c3c3f_389a_4cd0_8897_e870cea0e059.slice/crio-cb065c03759efdb4e212fdb6ff10e5d200bfad43b817e95554fececb1c89cffe WatchSource:0}: Error finding container cb065c03759efdb4e212fdb6ff10e5d200bfad43b817e95554fececb1c89cffe: Status 404 returned error can't find the container with id cb065c03759efdb4e212fdb6ff10e5d200bfad43b817e95554fececb1c89cffe Jul 06 22:32:51.241482 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.241442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g258p\" (UniqueName: \"kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p\") pod \"authorino-75d556bd77-5x4hp\" (UID: \"8f406ece-0409-4964-b2d2-1c097ae4c161\") " pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:32:51.251285 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.251253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g258p\" (UniqueName: \"kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p\") pod \"authorino-75d556bd77-5x4hp\" (UID: \"8f406ece-0409-4964-b2d2-1c097ae4c161\") " pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:32:51.264856 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.264803 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:32:51.422281 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.422242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:32:51.425566 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:32:51.425534 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f406ece_0409_4964_b2d2_1c097ae4c161.slice/crio-d68a03fee0e214e3f701350be51e3caed324edcdf2871e36806f486842c24f5b WatchSource:0}: Error finding container d68a03fee0e214e3f701350be51e3caed324edcdf2871e36806f486842c24f5b: Status 404 returned error can't find the container with id d68a03fee0e214e3f701350be51e3caed324edcdf2871e36806f486842c24f5b Jul 06 22:32:51.437930 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.437897 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qbdpf" event={"ID":"9d1c3c3f-389a-4cd0-8897-e870cea0e059","Type":"ContainerStarted","Data":"cb065c03759efdb4e212fdb6ff10e5d200bfad43b817e95554fececb1c89cffe"} Jul 06 22:32:51.439079 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:51.439048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5x4hp" event={"ID":"8f406ece-0409-4964-b2d2-1c097ae4c161","Type":"ContainerStarted","Data":"d68a03fee0e214e3f701350be51e3caed324edcdf2871e36806f486842c24f5b"} Jul 06 22:32:55.460018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:55.459911 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5x4hp" event={"ID":"8f406ece-0409-4964-b2d2-1c097ae4c161","Type":"ContainerStarted","Data":"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e"} Jul 06 22:32:55.461366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:55.461342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qbdpf" event={"ID":"9d1c3c3f-389a-4cd0-8897-e870cea0e059","Type":"ContainerStarted","Data":"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6"} Jul 06 22:32:55.475152 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:55.475101 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5x4hp" podStartSLOduration=2.250737453 podStartE2EDuration="5.475084s" totalImagesPullingTime="3.224346547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:55.474246703 +0000 UTC m=+531.500661068" watchObservedRunningTime="2026-07-06 22:32:55.475084 +0000 UTC m=+531.501498363" Jul 06 22:32:55.487845 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:55.487783 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qbdpf" podStartSLOduration=2.050969313 podStartE2EDuration="5.487760956s" totalImagesPullingTime="3.436791643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:55.487104276 +0000 UTC m=+531.513518638" watchObservedRunningTime="2026-07-06 22:32:55.487760956 +0000 UTC m=+531.514175321" Jul 06 22:32:55.511799 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:55.511766 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:57.469446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:57.469390 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qbdpf" podUID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" containerName="authorino" containerID="cri-o://cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6" gracePeriod=30 Jul 06 22:32:57.711303 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:57.711275 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:57.786308 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:57.786207 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts5qp\" (UniqueName: \"kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp\") pod \"9d1c3c3f-389a-4cd0-8897-e870cea0e059\" (UID: \"9d1c3c3f-389a-4cd0-8897-e870cea0e059\") " Jul 06 22:32:57.788421 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:57.788390 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp" (OuterVolumeSpecName: "kube-api-access-ts5qp") pod "9d1c3c3f-389a-4cd0-8897-e870cea0e059" (UID: "9d1c3c3f-389a-4cd0-8897-e870cea0e059"). InnerVolumeSpecName "kube-api-access-ts5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:32:57.887219 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:57.887186 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5qp\" (UniqueName: \"kubernetes.io/projected/9d1c3c3f-389a-4cd0-8897-e870cea0e059-kube-api-access-ts5qp\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:32:58.474660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.474620 2576 generic.go:356] "Generic (PLEG): container finished" podID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" containerID="cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6" exitCode=0 Jul 06 22:32:58.475094 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.474685 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qbdpf" Jul 06 22:32:58.475094 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.474710 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qbdpf" event={"ID":"9d1c3c3f-389a-4cd0-8897-e870cea0e059","Type":"ContainerDied","Data":"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6"} Jul 06 22:32:58.475094 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.474762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qbdpf" event={"ID":"9d1c3c3f-389a-4cd0-8897-e870cea0e059","Type":"ContainerDied","Data":"cb065c03759efdb4e212fdb6ff10e5d200bfad43b817e95554fececb1c89cffe"} Jul 06 22:32:58.475094 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.474779 2576 scope.go:117] "RemoveContainer" containerID="cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6" Jul 06 22:32:58.483264 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.483242 2576 scope.go:117] "RemoveContainer" containerID="cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6" Jul 06 22:32:58.483545 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:32:58.483522 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6\": container with ID starting with cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6 not found: ID does not exist" containerID="cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6" Jul 06 22:32:58.483602 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.483555 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6"} err="failed to get container status \"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6\": rpc error: code = NotFound desc = could not find container \"cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6\": container with ID starting with cc0b655b478f242b75bbcbde20cb0afb6362e4f1141d488d7c4daa2381c3cbc6 not found: ID does not exist" Jul 06 22:32:58.495024 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.494971 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:58.500528 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.500502 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qbdpf"] Jul 06 22:32:58.521039 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:32:58.521008 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" path="/var/lib/kubelet/pods/9d1c3c3f-389a-4cd0-8897-e870cea0e059/volumes" Jul 06 22:33:23.677160 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.677129 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9g687"] Jul 06 22:33:23.677704 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.677427 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" containerName="authorino" Jul 06 22:33:23.677704 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.677440 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" containerName="authorino" Jul 06 22:33:23.677704 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.677508 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1c3c3f-389a-4cd0-8897-e870cea0e059" containerName="authorino" Jul 06 22:33:23.682109 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.682089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:23.686569 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.686543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9g687"] Jul 06 22:33:23.779381 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.779344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvvw\" (UniqueName: \"kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw\") pod \"authorino-5d4b67c446-9g687\" (UID: \"950a779a-4c5c-427d-8bcc-61bf0104f2e8\") " pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:23.868603 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.868568 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9g687"] Jul 06 22:33:23.868813 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:33:23.868792 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ffvvw], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-9g687" podUID="950a779a-4c5c-427d-8bcc-61bf0104f2e8" Jul 06 22:33:23.880687 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.880655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvvw\" (UniqueName: \"kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw\") pod \"authorino-5d4b67c446-9g687\" (UID: \"950a779a-4c5c-427d-8bcc-61bf0104f2e8\") " pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:23.894162 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.894134 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvvw\" (UniqueName: \"kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw\") pod \"authorino-5d4b67c446-9g687\" (UID: \"950a779a-4c5c-427d-8bcc-61bf0104f2e8\") " pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:23.896678 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.896653 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-ngczm"] Jul 06 22:33:23.898660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.898644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:23.901191 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.901170 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 22:33:23.909767 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.909743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-ngczm"] Jul 06 22:33:23.936042 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.935939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-ngczm"] Jul 06 22:33:23.936211 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:33:23.936191 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vpxtt tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-vpxtt tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-ngczm" podUID="22700fdc-0db2-4209-baf6-b5ac4fafd318" Jul 06 22:33:23.963287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.963251 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:33:23.965440 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.965422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:23.974071 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.974046 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:33:23.982035 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.982009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:23.982182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.982065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:23.982182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.982084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4f9w\" (UniqueName: \"kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:23.982182 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:23.982142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxtt\" (UniqueName: \"kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.083307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.083265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:24.083307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.083308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4f9w\" (UniqueName: \"kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:24.083578 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.083339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxtt\" (UniqueName: \"kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.083578 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.083394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.085798 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.085771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.085908 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.085856 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:24.091209 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.091175 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxtt\" (UniqueName: \"kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt\") pod \"authorino-76877994-ngczm\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.091209 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.091208 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4f9w\" (UniqueName: \"kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w\") pod \"authorino-7b46b857cf-h2p8q\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:24.275110 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.275000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:33:24.399654 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.399619 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:33:24.402494 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:33:24.402470 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04562dd_bc46_4b16_a794_f2898b5b3700.slice/crio-e4b27b0e90e51ef4b82fbf1e1d520d8ba6edca3b5c2841c62e009da60da88913 WatchSource:0}: Error finding container e4b27b0e90e51ef4b82fbf1e1d520d8ba6edca3b5c2841c62e009da60da88913: Status 404 returned error can't find the container with id e4b27b0e90e51ef4b82fbf1e1d520d8ba6edca3b5c2841c62e009da60da88913 Jul 06 22:33:24.571691 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.571614 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.571838 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.571611 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" event={"ID":"b04562dd-bc46-4b16-a794-f2898b5b3700","Type":"ContainerStarted","Data":"e4b27b0e90e51ef4b82fbf1e1d520d8ba6edca3b5c2841c62e009da60da88913"} Jul 06 22:33:24.571838 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.571626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:24.578186 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.578161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:24.580454 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.580435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:24.688404 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.688374 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert\") pod \"22700fdc-0db2-4209-baf6-b5ac4fafd318\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " Jul 06 22:33:24.688404 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.688407 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpxtt\" (UniqueName: \"kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt\") pod \"22700fdc-0db2-4209-baf6-b5ac4fafd318\" (UID: \"22700fdc-0db2-4209-baf6-b5ac4fafd318\") " Jul 06 22:33:24.688826 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.688446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffvvw\" (UniqueName: \"kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw\") pod \"950a779a-4c5c-427d-8bcc-61bf0104f2e8\" (UID: \"950a779a-4c5c-427d-8bcc-61bf0104f2e8\") " Jul 06 22:33:24.690675 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.690647 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "22700fdc-0db2-4209-baf6-b5ac4fafd318" (UID: "22700fdc-0db2-4209-baf6-b5ac4fafd318"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:33:24.690776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.690689 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw" (OuterVolumeSpecName: "kube-api-access-ffvvw") pod "950a779a-4c5c-427d-8bcc-61bf0104f2e8" (UID: "950a779a-4c5c-427d-8bcc-61bf0104f2e8"). InnerVolumeSpecName "kube-api-access-ffvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:33:24.690776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.690707 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt" (OuterVolumeSpecName: "kube-api-access-vpxtt") pod "22700fdc-0db2-4209-baf6-b5ac4fafd318" (UID: "22700fdc-0db2-4209-baf6-b5ac4fafd318"). InnerVolumeSpecName "kube-api-access-vpxtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:33:24.788918 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.788892 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvvw\" (UniqueName: \"kubernetes.io/projected/950a779a-4c5c-427d-8bcc-61bf0104f2e8-kube-api-access-ffvvw\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:33:24.789028 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.788920 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22700fdc-0db2-4209-baf6-b5ac4fafd318-tls-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:33:24.789028 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:24.788936 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxtt\" (UniqueName: \"kubernetes.io/projected/22700fdc-0db2-4209-baf6-b5ac4fafd318-kube-api-access-vpxtt\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:33:25.576835 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.576800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" event={"ID":"b04562dd-bc46-4b16-a794-f2898b5b3700","Type":"ContainerStarted","Data":"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2"} Jul 06 22:33:25.577041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.576910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-ngczm" Jul 06 22:33:25.577041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.577028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9g687" Jul 06 22:33:25.593018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.592955 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" podStartSLOduration=2.229132935 podStartE2EDuration="2.59294037s" totalImagesPullingTime="363.807435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:33:25.592050034 +0000 UTC m=+561.618464399" watchObservedRunningTime="2026-07-06 22:33:25.59294037 +0000 UTC m=+561.619354733" Jul 06 22:33:25.615304 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.615271 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9g687"] Jul 06 22:33:25.621524 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.621498 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:33:25.621688 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.621667 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5x4hp" podUID="8f406ece-0409-4964-b2d2-1c097ae4c161" containerName="authorino" containerID="cri-o://24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e" gracePeriod=30 Jul 06 22:33:25.623450 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.623423 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9g687"] Jul 06 22:33:25.648053 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.648022 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-ngczm"] Jul 06 22:33:25.651617 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.651585 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-ngczm"] Jul 06 22:33:25.859608 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.859581 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:33:25.897666 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.897638 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g258p\" (UniqueName: \"kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p\") pod \"8f406ece-0409-4964-b2d2-1c097ae4c161\" (UID: \"8f406ece-0409-4964-b2d2-1c097ae4c161\") " Jul 06 22:33:25.899685 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.899658 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p" (OuterVolumeSpecName: "kube-api-access-g258p") pod "8f406ece-0409-4964-b2d2-1c097ae4c161" (UID: "8f406ece-0409-4964-b2d2-1c097ae4c161"). InnerVolumeSpecName "kube-api-access-g258p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:33:25.998263 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:25.998226 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g258p\" (UniqueName: \"kubernetes.io/projected/8f406ece-0409-4964-b2d2-1c097ae4c161-kube-api-access-g258p\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:33:26.521585 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.521551 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22700fdc-0db2-4209-baf6-b5ac4fafd318" path="/var/lib/kubelet/pods/22700fdc-0db2-4209-baf6-b5ac4fafd318/volumes" Jul 06 22:33:26.521799 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.521766 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950a779a-4c5c-427d-8bcc-61bf0104f2e8" path="/var/lib/kubelet/pods/950a779a-4c5c-427d-8bcc-61bf0104f2e8/volumes" Jul 06 22:33:26.582243 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.582210 2576 generic.go:356] "Generic (PLEG): container finished" podID="8f406ece-0409-4964-b2d2-1c097ae4c161" containerID="24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e" exitCode=0 Jul 06 22:33:26.582418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.582252 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5x4hp" Jul 06 22:33:26.582418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.582294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5x4hp" event={"ID":"8f406ece-0409-4964-b2d2-1c097ae4c161","Type":"ContainerDied","Data":"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e"} Jul 06 22:33:26.582418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.582331 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5x4hp" event={"ID":"8f406ece-0409-4964-b2d2-1c097ae4c161","Type":"ContainerDied","Data":"d68a03fee0e214e3f701350be51e3caed324edcdf2871e36806f486842c24f5b"} Jul 06 22:33:26.582418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.582349 2576 scope.go:117] "RemoveContainer" containerID="24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e" Jul 06 22:33:26.590292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.590270 2576 scope.go:117] "RemoveContainer" containerID="24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e" Jul 06 22:33:26.590548 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:33:26.590530 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e\": container with ID starting with 24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e not found: ID does not exist" containerID="24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e" Jul 06 22:33:26.590620 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.590555 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e"} err="failed to get container status \"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e\": rpc error: code = NotFound desc = could not find container \"24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e\": container with ID starting with 24d5adbab3920898386a3aece7cbdbadc43f48bf9f5ef06f1919ef5289d3162e not found: ID does not exist" Jul 06 22:33:26.596707 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.596687 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:33:26.598742 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:26.598722 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5x4hp"] Jul 06 22:33:28.520477 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:33:28.520444 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f406ece-0409-4964-b2d2-1c097ae4c161" path="/var/lib/kubelet/pods/8f406ece-0409-4964-b2d2-1c097ae4c161/volumes" Jul 06 22:34:03.391921 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.391880 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7mbw6"] Jul 06 22:34:03.392418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.392182 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f406ece-0409-4964-b2d2-1c097ae4c161" containerName="authorino" Jul 06 22:34:03.392418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.392194 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f406ece-0409-4964-b2d2-1c097ae4c161" containerName="authorino" Jul 06 22:34:03.392418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.392261 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f406ece-0409-4964-b2d2-1c097ae4c161" containerName="authorino" Jul 06 22:34:03.395308 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.395291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.397590 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.397567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jf5p4\"" Jul 06 22:34:03.397736 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.397602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 22:34:03.406268 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.406244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7mbw6"] Jul 06 22:34:03.492670 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.492633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf79\" (UniqueName: \"kubernetes.io/projected/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-kube-api-access-6jf79\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.492670 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.492681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.594067 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.594023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf79\" (UniqueName: \"kubernetes.io/projected/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-kube-api-access-6jf79\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.594273 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.594073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.594798 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.594771 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-plugins-config-volume\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.602261 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.602229 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf79\" (UniqueName: \"kubernetes.io/projected/9b0db4a9-e5be-4c04-9457-0ef3ff4e7275-kube-api-access-6jf79\") pod \"payload-pre-processing-7964457696-7mbw6\" (UID: \"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275\") " pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.705350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.705252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:03.835678 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:03.835643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-7mbw6"] Jul 06 22:34:03.838596 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:34:03.838568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0db4a9_e5be_4c04_9457_0ef3ff4e7275.slice/crio-ccd7e946dfb079fb4e4e55dbd6d9ff1290790b24cab0c7c2a3af982f194abe3a WatchSource:0}: Error finding container ccd7e946dfb079fb4e4e55dbd6d9ff1290790b24cab0c7c2a3af982f194abe3a: Status 404 returned error can't find the container with id ccd7e946dfb079fb4e4e55dbd6d9ff1290790b24cab0c7c2a3af982f194abe3a Jul 06 22:34:04.735648 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:04.735552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" event={"ID":"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275","Type":"ContainerStarted","Data":"ccd7e946dfb079fb4e4e55dbd6d9ff1290790b24cab0c7c2a3af982f194abe3a"} Jul 06 22:34:05.741310 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:05.741277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" event={"ID":"9b0db4a9-e5be-4c04-9457-0ef3ff4e7275","Type":"ContainerStarted","Data":"149350ff8d74bff5ca0625458e056c8928ff8fbc5fc1bccb48c3648230f9361c"} Jul 06 22:34:05.741699 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:05.741415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:05.757688 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:05.757634 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" podStartSLOduration=1.543495742 podStartE2EDuration="2.757618156s" totalImagesPullingTime="1.214122414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:05.757262856 +0000 UTC m=+601.783677211" watchObservedRunningTime="2026-07-06 22:34:05.757618156 +0000 UTC m=+601.784032519" Jul 06 22:34:16.748082 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:16.748043 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-7mbw6" Jul 06 22:34:58.367114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.367070 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f"] Jul 06 22:34:58.370591 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.370566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.373330 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.373301 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:34:58.373330 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.373302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 22:34:58.373542 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.373310 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p5kpm\"" Jul 06 22:34:58.374227 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.374208 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:34:58.380316 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.380290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f"] Jul 06 22:34:58.428664 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.428664 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e972134-ce83-4459-a5ba-d85be2b6a40d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.428878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.428878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.428878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6lsc\" (UniqueName: \"kubernetes.io/projected/1e972134-ce83-4459-a5ba-d85be2b6a40d-kube-api-access-q6lsc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.428878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.428823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.529904 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.529872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.529913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.529936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e972134-ce83-4459-a5ba-d85be2b6a40d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.529955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.530119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.530159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6lsc\" (UniqueName: \"kubernetes.io/projected/1e972134-ce83-4459-a5ba-d85be2b6a40d-kube-api-access-q6lsc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.530297 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530444 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.530361 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.530528 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.530453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.532299 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.532276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e972134-ce83-4459-a5ba-d85be2b6a40d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.532566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.532549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e972134-ce83-4459-a5ba-d85be2b6a40d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.538014 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.537957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6lsc\" (UniqueName: \"kubernetes.io/projected/1e972134-ce83-4459-a5ba-d85be2b6a40d-kube-api-access-q6lsc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q8b2f\" (UID: \"1e972134-ce83-4459-a5ba-d85be2b6a40d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.682934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.682893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:34:58.823435 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.823404 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f"] Jul 06 22:34:58.825912 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:34:58.825870 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e972134_ce83_4459_a5ba_d85be2b6a40d.slice/crio-678a07da1b7b7096f1d6b46c4f5f91204542bd0229018131f25877cb9234ee8a WatchSource:0}: Error finding container 678a07da1b7b7096f1d6b46c4f5f91204542bd0229018131f25877cb9234ee8a: Status 404 returned error can't find the container with id 678a07da1b7b7096f1d6b46c4f5f91204542bd0229018131f25877cb9234ee8a Jul 06 22:34:58.827338 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.827308 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:34:58.961217 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:34:58.961132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" event={"ID":"1e972134-ce83-4459-a5ba-d85be2b6a40d","Type":"ContainerStarted","Data":"678a07da1b7b7096f1d6b46c4f5f91204542bd0229018131f25877cb9234ee8a"} Jul 06 22:35:00.971057 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:00.971016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" event={"ID":"1e972134-ce83-4459-a5ba-d85be2b6a40d","Type":"ContainerStarted","Data":"e680ee914028b0778ff94cacd20ce29ccc8f485480048f7e17d8f0c135657d96"} Jul 06 22:35:00.990445 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:00.990388 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" podStartSLOduration=1.8889934529999999 podStartE2EDuration="2.990370843s" totalImagesPullingTime="1.10137739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:00.989458171 +0000 UTC m=+657.015872553" watchObservedRunningTime="2026-07-06 22:35:00.990370843 +0000 UTC m=+657.016785207" Jul 06 22:35:08.683615 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:08.683576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:35:08.684154 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:08.683644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:35:08.696257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:08.696225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:35:09.014462 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:09.014370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q8b2f" Jul 06 22:35:17.085426 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.085393 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9"] Jul 06 22:35:17.088732 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.088711 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.092242 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.092214 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 22:35:17.104372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.104337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9"] Jul 06 22:35:17.186169 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.186169 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.186395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.186395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.186395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59l7\" (UniqueName: \"kubernetes.io/projected/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-kube-api-access-d59l7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.186395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.186374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.287766 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.287724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59l7\" (UniqueName: \"kubernetes.io/projected/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-kube-api-access-d59l7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.287766 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.287768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288015 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.287901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288015 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.287939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288105 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.288019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288105 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.288051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288179 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.288166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288282 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.288261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.288372 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.288353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.290274 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.290246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.290570 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.290551 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.299174 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.299139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59l7\" (UniqueName: \"kubernetes.io/projected/c5ef5f2e-8037-420f-b911-1abbd7ce02b2-kube-api-access-d59l7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xkrf9\" (UID: \"c5ef5f2e-8037-420f-b911-1abbd7ce02b2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.399190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.399148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:17.535801 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:17.535775 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9"] Jul 06 22:35:17.538501 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:35:17.538469 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ef5f2e_8037_420f_b911_1abbd7ce02b2.slice/crio-1f7b51aaca32073ddedea75b66694e519458f86d9353d4bdee0c9066af90bf54 WatchSource:0}: Error finding container 1f7b51aaca32073ddedea75b66694e519458f86d9353d4bdee0c9066af90bf54: Status 404 returned error can't find the container with id 1f7b51aaca32073ddedea75b66694e519458f86d9353d4bdee0c9066af90bf54 Jul 06 22:35:18.041835 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:18.041738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" event={"ID":"c5ef5f2e-8037-420f-b911-1abbd7ce02b2","Type":"ContainerStarted","Data":"b0a58e2cca292052ed83efc9e116654c482d5712fb0cbdb59d625676b01cd21d"} Jul 06 22:35:18.041835 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:18.041794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" event={"ID":"c5ef5f2e-8037-420f-b911-1abbd7ce02b2","Type":"ContainerStarted","Data":"1f7b51aaca32073ddedea75b66694e519458f86d9353d4bdee0c9066af90bf54"} Jul 06 22:35:18.075943 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:18.075885 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" podStartSLOduration=0.893423363 podStartE2EDuration="1.075869962s" totalImagesPullingTime="182.446599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:18.075686761 +0000 UTC m=+674.102101126" watchObservedRunningTime="2026-07-06 22:35:18.075869962 +0000 UTC m=+674.102284330" Jul 06 22:35:27.399929 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:27.399885 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:27.400557 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:27.400019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:27.412348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:27.412322 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:28.097133 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:28.097100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xkrf9" Jul 06 22:35:51.970790 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:51.970746 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88"] Jul 06 22:35:51.976388 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:51.976359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:51.978970 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:51.978943 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 22:35:51.985641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:51.985614 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88"] Jul 06 22:35:52.073682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073641 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.073889 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsdv\" (UniqueName: \"kubernetes.io/projected/07c24cee-1577-4e97-acd7-6cb94c3340e4-kube-api-access-mqsdv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.073889 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c24cee-1577-4e97-acd7-6cb94c3340e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.073889 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.073889 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.074104 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.073931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.174752 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsdv\" (UniqueName: \"kubernetes.io/projected/07c24cee-1577-4e97-acd7-6cb94c3340e4-kube-api-access-mqsdv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.174752 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c24cee-1577-4e97-acd7-6cb94c3340e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.174961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175260 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.175245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175333 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.175312 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.175405 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.175377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.177100 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.177082 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07c24cee-1577-4e97-acd7-6cb94c3340e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.177470 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.177448 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07c24cee-1577-4e97-acd7-6cb94c3340e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.182602 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.182574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsdv\" (UniqueName: \"kubernetes.io/projected/07c24cee-1577-4e97-acd7-6cb94c3340e4-kube-api-access-mqsdv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88\" (UID: \"07c24cee-1577-4e97-acd7-6cb94c3340e4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.287052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.286930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:35:52.421205 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:52.421134 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88"] Jul 06 22:35:52.424839 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:35:52.424802 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07c24cee_1577_4e97_acd7_6cb94c3340e4.slice/crio-82aba01e9209d1f40499069f427427ed57fd75fad67041cd2168edbfbb543048 WatchSource:0}: Error finding container 82aba01e9209d1f40499069f427427ed57fd75fad67041cd2168edbfbb543048: Status 404 returned error can't find the container with id 82aba01e9209d1f40499069f427427ed57fd75fad67041cd2168edbfbb543048 Jul 06 22:35:53.191247 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:53.191202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" event={"ID":"07c24cee-1577-4e97-acd7-6cb94c3340e4","Type":"ContainerStarted","Data":"6158c776b91d021b3a418d1e6a1f9990a0dcf2721e7dc202bf0685c2de6a4a49"} Jul 06 22:35:53.191247 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:53.191254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" event={"ID":"07c24cee-1577-4e97-acd7-6cb94c3340e4","Type":"ContainerStarted","Data":"82aba01e9209d1f40499069f427427ed57fd75fad67041cd2168edbfbb543048"} Jul 06 22:35:53.208125 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:35:53.208071 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" podStartSLOduration=2.026253923 podStartE2EDuration="2.208053443s" totalImagesPullingTime="181.79952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:53.206879627 +0000 UTC m=+709.233294002" watchObservedRunningTime="2026-07-06 22:35:53.208053443 +0000 UTC m=+709.234467806" Jul 06 22:36:02.287376 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:02.287330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:36:02.287376 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:02.287387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:36:02.299799 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:02.299772 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:36:03.243562 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:03.243534 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88" Jul 06 22:36:54.927178 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:54.927145 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77ff4d58f8-cv6w6"] Jul 06 22:36:54.930353 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:54.930329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:54.938244 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:54.937900 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77ff4d58f8-cv6w6"] Jul 06 22:36:55.008039 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.008001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b765adc-3a35-4595-a134-52eeb7b5d194-tls-cert\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.008232 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.008087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjpw\" (UniqueName: \"kubernetes.io/projected/7b765adc-3a35-4595-a134-52eeb7b5d194-kube-api-access-wsjpw\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.109202 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.109157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b765adc-3a35-4595-a134-52eeb7b5d194-tls-cert\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.109381 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.109232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjpw\" (UniqueName: \"kubernetes.io/projected/7b765adc-3a35-4595-a134-52eeb7b5d194-kube-api-access-wsjpw\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.111861 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.111837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b765adc-3a35-4595-a134-52eeb7b5d194-tls-cert\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.117736 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.117704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjpw\" (UniqueName: \"kubernetes.io/projected/7b765adc-3a35-4595-a134-52eeb7b5d194-kube-api-access-wsjpw\") pod \"authorino-77ff4d58f8-cv6w6\" (UID: \"7b765adc-3a35-4595-a134-52eeb7b5d194\") " pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.240910 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.240817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" Jul 06 22:36:55.381277 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.381236 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77ff4d58f8-cv6w6"] Jul 06 22:36:55.384360 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:36:55.384331 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b765adc_3a35_4595_a134_52eeb7b5d194.slice/crio-f800bde9ea53a810a72956f4b17f88663bc9ca795d812e798d614b76391bd44e WatchSource:0}: Error finding container f800bde9ea53a810a72956f4b17f88663bc9ca795d812e798d614b76391bd44e: Status 404 returned error can't find the container with id f800bde9ea53a810a72956f4b17f88663bc9ca795d812e798d614b76391bd44e Jul 06 22:36:55.448998 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:55.448942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" event={"ID":"7b765adc-3a35-4595-a134-52eeb7b5d194","Type":"ContainerStarted","Data":"f800bde9ea53a810a72956f4b17f88663bc9ca795d812e798d614b76391bd44e"} Jul 06 22:36:56.455509 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.455465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" event={"ID":"7b765adc-3a35-4595-a134-52eeb7b5d194","Type":"ContainerStarted","Data":"a99f298c5148684b89a89ded8fca4db4b003754f5c3fe2aee54289b978ae4acf"} Jul 06 22:36:56.476270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.476213 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77ff4d58f8-cv6w6" podStartSLOduration=1.964227319 podStartE2EDuration="2.476184077s" totalImagesPullingTime="511.956758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:36:56.475235424 +0000 UTC m=+772.501649790" watchObservedRunningTime="2026-07-06 22:36:56.476184077 +0000 UTC m=+772.502598440" Jul 06 22:36:56.508268 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.508227 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:36:56.509040 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.508575 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" podUID="b04562dd-bc46-4b16-a794-f2898b5b3700" containerName="authorino" containerID="cri-o://7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2" gracePeriod=30 Jul 06 22:36:56.778140 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.778105 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:36:56.924783 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.924739 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert\") pod \"b04562dd-bc46-4b16-a794-f2898b5b3700\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " Jul 06 22:36:56.924934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.924826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4f9w\" (UniqueName: \"kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w\") pod \"b04562dd-bc46-4b16-a794-f2898b5b3700\" (UID: \"b04562dd-bc46-4b16-a794-f2898b5b3700\") " Jul 06 22:36:56.926972 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.926933 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w" (OuterVolumeSpecName: "kube-api-access-t4f9w") pod "b04562dd-bc46-4b16-a794-f2898b5b3700" (UID: "b04562dd-bc46-4b16-a794-f2898b5b3700"). InnerVolumeSpecName "kube-api-access-t4f9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:36:56.935634 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:56.935588 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b04562dd-bc46-4b16-a794-f2898b5b3700" (UID: "b04562dd-bc46-4b16-a794-f2898b5b3700"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:36:57.025682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.025588 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b04562dd-bc46-4b16-a794-f2898b5b3700-tls-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:36:57.025682 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.025631 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4f9w\" (UniqueName: \"kubernetes.io/projected/b04562dd-bc46-4b16-a794-f2898b5b3700-kube-api-access-t4f9w\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:36:57.461061 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.461024 2576 generic.go:356] "Generic (PLEG): container finished" podID="b04562dd-bc46-4b16-a794-f2898b5b3700" containerID="7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2" exitCode=0 Jul 06 22:36:57.461549 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.461069 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" Jul 06 22:36:57.461549 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.461110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" event={"ID":"b04562dd-bc46-4b16-a794-f2898b5b3700","Type":"ContainerDied","Data":"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2"} Jul 06 22:36:57.461549 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.461163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46b857cf-h2p8q" event={"ID":"b04562dd-bc46-4b16-a794-f2898b5b3700","Type":"ContainerDied","Data":"e4b27b0e90e51ef4b82fbf1e1d520d8ba6edca3b5c2841c62e009da60da88913"} Jul 06 22:36:57.461549 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.461193 2576 scope.go:117] "RemoveContainer" containerID="7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2" Jul 06 22:36:57.470761 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.470735 2576 scope.go:117] "RemoveContainer" containerID="7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2" Jul 06 22:36:57.471220 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:36:57.471192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2\": container with ID starting with 7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2 not found: ID does not exist" containerID="7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2" Jul 06 22:36:57.471318 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.471234 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2"} err="failed to get container status \"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2\": rpc error: code = NotFound desc = could not find container \"7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2\": container with ID starting with 7b5b6720cb208a562fbf2787a42adf34f47f5b834098b87914fa1b89162c67a2 not found: ID does not exist" Jul 06 22:36:57.482689 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.482648 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:36:57.489168 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:57.489135 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b46b857cf-h2p8q"] Jul 06 22:36:58.521965 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:36:58.521921 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04562dd-bc46-4b16-a794-f2898b5b3700" path="/var/lib/kubelet/pods/b04562dd-bc46-4b16-a794-f2898b5b3700/volumes" Jul 06 22:50:16.750072 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.750027 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:16.752340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.750530 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04562dd-bc46-4b16-a794-f2898b5b3700" containerName="authorino" Jul 06 22:50:16.752340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.750549 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04562dd-bc46-4b16-a794-f2898b5b3700" containerName="authorino" Jul 06 22:50:16.752340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.750665 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04562dd-bc46-4b16-a794-f2898b5b3700" containerName="authorino" Jul 06 22:50:16.753730 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.753709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.756237 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.756218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ba18ff93-openshift-default-dockercfg-hjksn\"" Jul 06 22:50:16.768421 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.768391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:16.796405 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9dv7\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796801 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.796801 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.796628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897391 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9dv7\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897391 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.897641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.897954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898230 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.898089 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898230 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.898123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.898411 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.898391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.899853 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.899827 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.900138 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.900120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.905333 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.905307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:16.905475 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:16.905457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9dv7\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7\") pod \"e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:17.064618 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.064533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:17.205523 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.205217 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:17.208118 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:50:17.208090 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6548b443_7e0a_46c3_827f_0531840a2f01.slice/crio-4c50452d76633f6570b923d5615757c1ccaaf03c33c38ddc3d59969b02634443 WatchSource:0}: Error finding container 4c50452d76633f6570b923d5615757c1ccaaf03c33c38ddc3d59969b02634443: Status 404 returned error can't find the container with id 4c50452d76633f6570b923d5615757c1ccaaf03c33c38ddc3d59969b02634443 Jul 06 22:50:17.210391 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.210357 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:17.210469 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.210427 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:17.210469 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.210456 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:17.829772 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.829738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" event={"ID":"6548b443-7e0a-46c3-827f-0531840a2f01","Type":"ContainerStarted","Data":"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225"} Jul 06 22:50:17.830205 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.829786 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" event={"ID":"6548b443-7e0a-46c3-827f-0531840a2f01","Type":"ContainerStarted","Data":"4c50452d76633f6570b923d5615757c1ccaaf03c33c38ddc3d59969b02634443"} Jul 06 22:50:17.849581 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:17.849534 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" podStartSLOduration=1.849515465 podStartE2EDuration="1.849515465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:50:17.847911409 +0000 UTC m=+1573.874325776" watchObservedRunningTime="2026-07-06 22:50:17.849515465 +0000 UTC m=+1573.875929828" Jul 06 22:50:18.065788 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:18.065753 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:18.070704 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:18.070674 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:18.833892 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:18.833859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:18.834839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:18.834820 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:23.168685 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:23.168637 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:23.169236 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:23.168867 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" podUID="6548b443-7e0a-46c3-827f-0531840a2f01" containerName="istio-proxy" containerID="cri-o://82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225" gracePeriod=30 Jul 06 22:50:29.032748 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.032721 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:29.121215 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121128 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9dv7\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121215 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121215 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121213 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121228 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121287 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121377 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data\") pod \"6548b443-7e0a-46c3-827f-0531840a2f01\" (UID: \"6548b443-7e0a-46c3-827f-0531840a2f01\") " Jul 06 22:50:29.121781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121606 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:29.121781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121618 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:29.121781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121736 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:29.121934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.121793 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:50:29.122256 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.122236 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.122356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.122261 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6548b443-7e0a-46c3-827f-0531840a2f01-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.122356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.122277 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.122356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.122290 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.122482 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.122415 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data" (OuterVolumeSpecName: "istio-data") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:29.123746 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.123720 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:50:29.123855 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.123814 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token" (OuterVolumeSpecName: "istio-token") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:29.123912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.123849 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7" (OuterVolumeSpecName: "kube-api-access-t9dv7") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "kube-api-access-t9dv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:29.123912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.123849 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6548b443-7e0a-46c3-827f-0531840a2f01" (UID: "6548b443-7e0a-46c3-827f-0531840a2f01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:29.222834 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.222788 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6548b443-7e0a-46c3-827f-0531840a2f01-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.222834 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.222820 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.222834 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.222832 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9dv7\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-kube-api-access-t9dv7\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.222834 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.222842 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6548b443-7e0a-46c3-827f-0531840a2f01-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.222834 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.222849 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6548b443-7e0a-46c3-827f-0531840a2f01-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:29.333784 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.333744 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:29.334245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.334229 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6548b443-7e0a-46c3-827f-0531840a2f01" containerName="istio-proxy" Jul 06 22:50:29.334297 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.334249 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6548b443-7e0a-46c3-827f-0531840a2f01" containerName="istio-proxy" Jul 06 22:50:29.334331 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.334320 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6548b443-7e0a-46c3-827f-0531840a2f01" containerName="istio-proxy" Jul 06 22:50:29.336925 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.336900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.339543 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.339522 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6342cf76-openshift-default-dockercfg-lr6cx\"" Jul 06 22:50:29.351679 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.351643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:29.426095 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426095 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz575\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426259 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.426340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.426317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527074 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527074 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527077 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz575\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527317 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527583 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527583 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527583 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527550 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527583 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.527786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.527770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.528090 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.528071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.529508 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.529491 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.529579 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.529568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.541616 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.541587 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.541816 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.541796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz575\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575\") pod \"e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.647905 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.647872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:29.783575 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.783541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:29.788224 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:50:29.788189 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cf600d_3356_4f95_bff1_4cc44a51d8ad.slice/crio-7a8843186cf86fdc37ea3cbba986b7721ee0dc78f45c1380bc513037a20350a1 WatchSource:0}: Error finding container 7a8843186cf86fdc37ea3cbba986b7721ee0dc78f45c1380bc513037a20350a1: Status 404 returned error can't find the container with id 7a8843186cf86fdc37ea3cbba986b7721ee0dc78f45c1380bc513037a20350a1 Jul 06 22:50:29.790231 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.790202 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:29.790321 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.790270 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:29.790321 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.790299 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:29.887260 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.887224 2576 generic.go:356] "Generic (PLEG): container finished" podID="6548b443-7e0a-46c3-827f-0531840a2f01" containerID="82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225" exitCode=0 Jul 06 22:50:29.887260 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.887265 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" event={"ID":"6548b443-7e0a-46c3-827f-0531840a2f01","Type":"ContainerDied","Data":"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225"} Jul 06 22:50:29.887525 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.887325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" event={"ID":"6548b443-7e0a-46c3-827f-0531840a2f01","Type":"ContainerDied","Data":"4c50452d76633f6570b923d5615757c1ccaaf03c33c38ddc3d59969b02634443"} Jul 06 22:50:29.887525 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.887272 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr" Jul 06 22:50:29.887525 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.887355 2576 scope.go:117] "RemoveContainer" containerID="82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225" Jul 06 22:50:29.889222 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.889191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerStarted","Data":"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878"} Jul 06 22:50:29.889345 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.889245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerStarted","Data":"7a8843186cf86fdc37ea3cbba986b7721ee0dc78f45c1380bc513037a20350a1"} Jul 06 22:50:29.897119 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.896856 2576 scope.go:117] "RemoveContainer" containerID="82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225" Jul 06 22:50:29.897259 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:50:29.897233 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225\": container with ID starting with 82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225 not found: ID does not exist" containerID="82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225" Jul 06 22:50:29.897339 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.897270 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225"} err="failed to get container status \"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225\": rpc error: code = NotFound desc = could not find container \"82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225\": container with ID starting with 82bece3e1425284a5af7765403941529d748a3253f554836e0451a3e5faaf225 not found: ID does not exist" Jul 06 22:50:29.911096 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.911045 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" podStartSLOduration=0.911030337 podStartE2EDuration="911.030337ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:50:29.909067967 +0000 UTC m=+1585.935482331" watchObservedRunningTime="2026-07-06 22:50:29.911030337 +0000 UTC m=+1585.937444701" Jul 06 22:50:29.921494 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.921463 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:29.925235 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:29.925198 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ba18ff93-openshift-default-54648c97bc-tlxfr"] Jul 06 22:50:30.522128 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:30.522096 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6548b443-7e0a-46c3-827f-0531840a2f01" path="/var/lib/kubelet/pods/6548b443-7e0a-46c3-827f-0531840a2f01/volumes" Jul 06 22:50:30.648129 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:30.648100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:30.653128 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:30.653095 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:30.895109 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:30.895080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:30.896162 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:30.896143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:40.941209 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.941122 2576 generic.go:356] "Generic (PLEG): container finished" podID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerID="6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878" exitCode=0 Jul 06 22:50:40.941209 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.941194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerDied","Data":"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878"} Jul 06 22:50:40.941658 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.941588 2576 scope.go:117] "RemoveContainer" containerID="6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878" Jul 06 22:50:40.942678 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.942637 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:40.942772 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.942726 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:40.942824 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:40.942808 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:41.689786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:41.689746 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:41.947428 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:41.947329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerStarted","Data":"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223"} Jul 06 22:50:41.947855 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:41.947443 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" containerID="cri-o://83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223" gracePeriod=30 Jul 06 22:50:45.896044 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:45.896008 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:47.209940 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.209874 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:47.286163 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286118 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz575\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286174 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286316 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286514 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286372 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286514 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286410 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286514 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286514 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286485 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286706 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286526 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:47.286706 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286534 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket\") pod \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\" (UID: \"76cf600d-3356-4f95-bff1-4cc44a51d8ad\") " Jul 06 22:50:47.286706 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286581 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:47.286861 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286763 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:47.286916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286899 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.287003 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286923 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.287003 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.286938 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.287110 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.287017 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:50:47.287519 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.287499 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:47.288539 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.288507 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:47.288611 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.288578 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:47.288733 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.288713 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:50:47.288832 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.288812 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575" (OuterVolumeSpecName: "kube-api-access-dz575") pod "76cf600d-3356-4f95-bff1-4cc44a51d8ad" (UID: "76cf600d-3356-4f95-bff1-4cc44a51d8ad"). InnerVolumeSpecName "kube-api-access-dz575". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:47.388020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.387957 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz575\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-kube-api-access-dz575\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.388020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.388010 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.388020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.388021 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.388020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.388029 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.388271 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.388040 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.388271 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.388047 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76cf600d-3356-4f95-bff1-4cc44a51d8ad-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:47.946539 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.946504 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:50:47.946950 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.946936 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:50:47.947007 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.946954 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:50:47.947086 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.947072 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:50:47.947121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.947090 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:50:47.950533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.950514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.952969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.952947 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6cd14b60-openshift-default-dockercfg-l5np6\"" Jul 06 22:50:47.960630 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.960603 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:50:47.973638 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.973602 2576 generic.go:356] "Generic (PLEG): container finished" podID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerID="83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223" exitCode=0 Jul 06 22:50:47.973808 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.973670 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerDied","Data":"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223"} Jul 06 22:50:47.973808 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.973722 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" event={"ID":"76cf600d-3356-4f95-bff1-4cc44a51d8ad","Type":"ContainerDied","Data":"7a8843186cf86fdc37ea3cbba986b7721ee0dc78f45c1380bc513037a20350a1"} Jul 06 22:50:47.973808 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.973674 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr" Jul 06 22:50:47.973968 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.973751 2576 scope.go:117] "RemoveContainer" containerID="83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223" Jul 06 22:50:47.985633 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.985612 2576 scope.go:117] "RemoveContainer" containerID="6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878" Jul 06 22:50:47.992786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.992916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.992916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.992916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.992916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.993151 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.992968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7z92\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.993151 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.993040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.993151 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.993080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.993309 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.993188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:47.995071 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.995048 2576 scope.go:117] "RemoveContainer" containerID="83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223" Jul 06 22:50:47.995383 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:50:47.995366 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223\": container with ID starting with 83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223 not found: ID does not exist" containerID="83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223" Jul 06 22:50:47.995454 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.995391 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223"} err="failed to get container status \"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223\": rpc error: code = NotFound desc = could not find container \"83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223\": container with ID starting with 83872d1e3d8fede35281449635fc1aff2c757fa72b31e15df18ead1717f85223 not found: ID does not exist" Jul 06 22:50:47.995454 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.995408 2576 scope.go:117] "RemoveContainer" containerID="6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878" Jul 06 22:50:47.995668 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:50:47.995652 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878\": container with ID starting with 6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878 not found: ID does not exist" containerID="6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878" Jul 06 22:50:47.995711 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:47.995672 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878"} err="failed to get container status \"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878\": rpc error: code = NotFound desc = could not find container \"6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878\": container with ID starting with 6fda0bcd59714f2b14d456d79283bb691887805e92e53c17afcc3671841f1878 not found: ID does not exist" Jul 06 22:50:48.003781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.003754 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:48.007392 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.007366 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6342cf76-openshift-default-867c7b5b85-67tqr"] Jul 06 22:50:48.093729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7z92\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.093916 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094295 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.093932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094295 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.094266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094400 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.094346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094542 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.094522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094606 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.094565 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.094731 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.094712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.096307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.096284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.096540 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.096519 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.101632 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.101602 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7z92\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.101743 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.101633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token\") pod \"e2e-derive-6cd14b60-openshift-default-7644b67bc-62282\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.262020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.261898 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:48.393624 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.393598 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:50:48.395725 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:50:48.395697 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f31e8a_8389_486c_8efc_1fc268002a3d.slice/crio-af8135df047281b150ac970287a693a5ab42f65bee0bce82cb7ed54330693b6c WatchSource:0}: Error finding container af8135df047281b150ac970287a693a5ab42f65bee0bce82cb7ed54330693b6c: Status 404 returned error can't find the container with id af8135df047281b150ac970287a693a5ab42f65bee0bce82cb7ed54330693b6c Jul 06 22:50:48.397792 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.397758 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:48.397869 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.397826 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:48.397869 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.397852 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:50:48.530797 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.530713 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" path="/var/lib/kubelet/pods/76cf600d-3356-4f95-bff1-4cc44a51d8ad/volumes" Jul 06 22:50:48.979746 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.979713 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" event={"ID":"61f31e8a-8389-486c-8efc-1fc268002a3d","Type":"ContainerStarted","Data":"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa"} Jul 06 22:50:48.979921 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.979763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" event={"ID":"61f31e8a-8389-486c-8efc-1fc268002a3d","Type":"ContainerStarted","Data":"af8135df047281b150ac970287a693a5ab42f65bee0bce82cb7ed54330693b6c"} Jul 06 22:50:48.999526 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:48.999475 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" podStartSLOduration=1.999459329 podStartE2EDuration="1.999459329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:50:48.997807542 +0000 UTC m=+1605.024221906" watchObservedRunningTime="2026-07-06 22:50:48.999459329 +0000 UTC m=+1605.025873693" Jul 06 22:50:49.262330 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:49.262237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:49.267145 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:49.267117 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:49.985510 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:49.985468 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:49.986770 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:49.986751 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:53.705439 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:53.705389 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:50:53.705849 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:53.705588 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" podUID="61f31e8a-8389-486c-8efc-1fc268002a3d" containerName="istio-proxy" containerID="cri-o://196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa" gracePeriod=30 Jul 06 22:50:58.963056 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:58.963026 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:59.024024 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.023917 2576 generic.go:356] "Generic (PLEG): container finished" podID="61f31e8a-8389-486c-8efc-1fc268002a3d" containerID="196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa" exitCode=0 Jul 06 22:50:59.024024 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.023964 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" Jul 06 22:50:59.024024 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.024007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" event={"ID":"61f31e8a-8389-486c-8efc-1fc268002a3d","Type":"ContainerDied","Data":"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa"} Jul 06 22:50:59.024245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.024059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282" event={"ID":"61f31e8a-8389-486c-8efc-1fc268002a3d","Type":"ContainerDied","Data":"af8135df047281b150ac970287a693a5ab42f65bee0bce82cb7ed54330693b6c"} Jul 06 22:50:59.024245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.024082 2576 scope.go:117] "RemoveContainer" containerID="196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa" Jul 06 22:50:59.032363 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.032340 2576 scope.go:117] "RemoveContainer" containerID="196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa" Jul 06 22:50:59.032608 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:50:59.032589 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa\": container with ID starting with 196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa not found: ID does not exist" containerID="196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa" Jul 06 22:50:59.032672 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.032617 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa"} err="failed to get container status \"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa\": rpc error: code = NotFound desc = could not find container \"196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa\": container with ID starting with 196557eae99301a7ce674cc6791b7c9f0e407c1ed743b9f2b718cf7c763070fa not found: ID does not exist" Jul 06 22:50:59.086132 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086094 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086132 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086163 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7z92\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086180 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086232 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086252 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086280 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086332 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086322 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo\") pod \"61f31e8a-8389-486c-8efc-1fc268002a3d\" (UID: \"61f31e8a-8389-486c-8efc-1fc268002a3d\") " Jul 06 22:50:59.086661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086459 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:59.086661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086558 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.086661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086468 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:59.086661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086634 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:59.086829 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086661 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:59.086829 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.086744 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:50:59.088564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.088532 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:50:59.088671 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.088588 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92" (OuterVolumeSpecName: "kube-api-access-g7z92") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "kube-api-access-g7z92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:59.088671 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.088617 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:50:59.088671 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.088641 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61f31e8a-8389-486c-8efc-1fc268002a3d" (UID: "61f31e8a-8389-486c-8efc-1fc268002a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:50:59.187446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187407 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187436 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f31e8a-8389-486c-8efc-1fc268002a3d-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187445 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187453 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187461 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f31e8a-8389-486c-8efc-1fc268002a3d-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187469 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187478 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7z92\" (UniqueName: \"kubernetes.io/projected/61f31e8a-8389-486c-8efc-1fc268002a3d-kube-api-access-g7z92\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.187708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.187487 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f31e8a-8389-486c-8efc-1fc268002a3d-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:50:59.344453 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.344422 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:50:59.348586 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:50:59.348563 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6cd14b60-openshift-default-7644b67bc-62282"] Jul 06 22:51:00.522175 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:51:00.522142 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f31e8a-8389-486c-8efc-1fc268002a3d" path="/var/lib/kubelet/pods/61f31e8a-8389-486c-8efc-1fc268002a3d/volumes" Jul 06 22:52:36.706917 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.706823 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:52:36.707600 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.707290 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f31e8a-8389-486c-8efc-1fc268002a3d" containerName="istio-proxy" Jul 06 22:52:36.707600 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.707305 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f31e8a-8389-486c-8efc-1fc268002a3d" containerName="istio-proxy" Jul 06 22:52:36.707600 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.707335 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:52:36.707600 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.707343 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cf600d-3356-4f95-bff1-4cc44a51d8ad" containerName="istio-proxy" Jul 06 22:52:36.707600 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.707415 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f31e8a-8389-486c-8efc-1fc268002a3d" containerName="istio-proxy" Jul 06 22:52:36.710497 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.710476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.713878 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.713849 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7d7e43b3-openshift-default-dockercfg-jp82r\"" Jul 06 22:52:36.795604 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.795555 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:52:36.818477 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818666 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818666 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818666 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818666 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818883 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818883 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818883 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.818883 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.818816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw48v\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw48v\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919442 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.919940 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919892 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.920016 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.919955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.920016 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.920004 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.920131 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.920109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.920270 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.920242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.922047 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.922028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.922367 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.922346 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.927900 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.927879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:36.928100 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:36.928076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw48v\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v\") pod \"e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:37.021487 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.021388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:37.161055 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.161029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:52:37.163643 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:52:37.163597 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1bf9fd_a717_4299_a5b8_f1c08107256d.slice/crio-91653a5b92307b367832e4205360793574a4ccb92f531a304f5a7e8961bc6c8b WatchSource:0}: Error finding container 91653a5b92307b367832e4205360793574a4ccb92f531a304f5a7e8961bc6c8b: Status 404 returned error can't find the container with id 91653a5b92307b367832e4205360793574a4ccb92f531a304f5a7e8961bc6c8b Jul 06 22:52:37.165785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.165749 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:37.165873 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.165820 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:37.165873 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.165847 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:37.430133 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.430095 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerStarted","Data":"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3"} Jul 06 22:52:37.430308 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.430166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerStarted","Data":"91653a5b92307b367832e4205360793574a4ccb92f531a304f5a7e8961bc6c8b"} Jul 06 22:52:37.450968 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:37.450908 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podStartSLOduration=1.450890018 podStartE2EDuration="1.450890018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:37.448798747 +0000 UTC m=+1713.475213122" watchObservedRunningTime="2026-07-06 22:52:37.450890018 +0000 UTC m=+1713.477304382" Jul 06 22:52:38.022451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:38.022413 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:38.027312 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:38.027286 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:38.435635 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:38.435607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:38.436473 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:38.436446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:52.499645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.499612 2576 generic.go:356] "Generic (PLEG): container finished" podID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerID="0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3" exitCode=0 Jul 06 22:52:52.500106 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.499685 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerDied","Data":"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3"} Jul 06 22:52:52.500106 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.500095 2576 scope.go:117] "RemoveContainer" containerID="0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3" Jul 06 22:52:52.501207 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.501174 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:52.501337 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.501263 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:52.501337 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:52.501313 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:52:53.436025 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:53.435971 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:53.505386 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:53.505352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerStarted","Data":"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a"} Jul 06 22:52:54.022618 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:54.022572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:52:54.023704 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:54.023680 2576 patch_prober.go:28] interesting pod/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 06 22:52:54.023764 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:54.023731 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 06 22:52:55.022254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:55.022217 2576 patch_prober.go:28] interesting pod/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 06 22:52:55.022702 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:55.022283 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 06 22:52:56.022063 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:56.022026 2576 patch_prober.go:28] interesting pod/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 06 22:52:56.022249 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:56.022082 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 06 22:52:57.022418 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:57.022376 2576 patch_prober.go:28] interesting pod/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 06 22:52:57.022836 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:57.022471 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 06 22:52:57.709611 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:57.709570 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:52:57.709862 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:52:57.709811 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" containerID="cri-o://d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a" gracePeriod=30 Jul 06 22:53:03.269249 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.269220 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:53:03.353299 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353299 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353245 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw48v\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353317 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353344 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353393 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353434 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353546 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353529 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo\") pod \"de1bf9fd-a717-4299-a5b8-f1c08107256d\" (UID: \"de1bf9fd-a717-4299-a5b8-f1c08107256d\") " Jul 06 22:53:03.353898 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353620 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:53:03.353898 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353792 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.353898 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353799 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:53:03.353898 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353861 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:53:03.353898 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.353888 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:53:03.354143 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.354100 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data" (OuterVolumeSpecName: "istio-data") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:53:03.355683 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.355653 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v" (OuterVolumeSpecName: "kube-api-access-pw48v") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "kube-api-access-pw48v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:53:03.355683 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.355651 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:53:03.355863 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.355724 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token" (OuterVolumeSpecName: "istio-token") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:53:03.355863 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.355739 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de1bf9fd-a717-4299-a5b8-f1c08107256d" (UID: "de1bf9fd-a717-4299-a5b8-f1c08107256d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:53:03.454366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454326 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de1bf9fd-a717-4299-a5b8-f1c08107256d-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454355 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454365 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454374 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454626 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454383 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454626 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454392 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454626 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454399 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de1bf9fd-a717-4299-a5b8-f1c08107256d-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.454626 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.454409 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw48v\" (UniqueName: \"kubernetes.io/projected/de1bf9fd-a717-4299-a5b8-f1c08107256d-kube-api-access-pw48v\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:53:03.552229 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.552193 2576 generic.go:356] "Generic (PLEG): container finished" podID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerID="d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a" exitCode=0 Jul 06 22:53:03.552343 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.552239 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" Jul 06 22:53:03.552343 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.552243 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerDied","Data":"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a"} Jul 06 22:53:03.552343 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.552295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg" event={"ID":"de1bf9fd-a717-4299-a5b8-f1c08107256d","Type":"ContainerDied","Data":"91653a5b92307b367832e4205360793574a4ccb92f531a304f5a7e8961bc6c8b"} Jul 06 22:53:03.552343 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.552317 2576 scope.go:117] "RemoveContainer" containerID="d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a" Jul 06 22:53:03.561360 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.561339 2576 scope.go:117] "RemoveContainer" containerID="0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3" Jul 06 22:53:03.569197 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.569176 2576 scope.go:117] "RemoveContainer" containerID="d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a" Jul 06 22:53:03.569483 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:53:03.569460 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a\": container with ID starting with d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a not found: ID does not exist" containerID="d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a" Jul 06 22:53:03.569528 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.569492 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a"} err="failed to get container status \"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a\": rpc error: code = NotFound desc = could not find container \"d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a\": container with ID starting with d55b555f9be7c5c22c0cda06fc5b9fda357305022dd24f27107272b11d6d836a not found: ID does not exist" Jul 06 22:53:03.569528 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.569510 2576 scope.go:117] "RemoveContainer" containerID="0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3" Jul 06 22:53:03.569717 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:53:03.569700 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3\": container with ID starting with 0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3 not found: ID does not exist" containerID="0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3" Jul 06 22:53:03.569754 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.569721 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3"} err="failed to get container status \"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3\": rpc error: code = NotFound desc = could not find container \"0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3\": container with ID starting with 0bfb11174639cbf8ae2b1c01a153baf4807fdc16492e148728b31448982c34e3 not found: ID does not exist" Jul 06 22:53:03.573550 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.573522 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:53:03.577034 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:03.577010 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7d7e43b3-openshift-default-f8d8ffcff-7tnsg"] Jul 06 22:53:04.522341 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:04.522310 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" path="/var/lib/kubelet/pods/de1bf9fd-a717-4299-a5b8-f1c08107256d/volumes" Jul 06 22:53:17.856893 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.856857 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857191 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857204 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857226 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857233 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857320 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.857786 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.857331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1bf9fd-a717-4299-a5b8-f1c08107256d" containerName="istio-proxy" Jul 06 22:53:17.860328 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.860298 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.862875 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.862840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-75c546e0-openshift-default-dockercfg-szh56\"" Jul 06 22:53:17.874211 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.874183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:53:17.972601 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972698 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.972813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkrg\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.973143 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:17.973143 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:17.972884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074501 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.074708 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkrg\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.075047 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.075047 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.074767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.075245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.075219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.075725 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.075703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.075787 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.075751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.076142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.076115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.076411 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.076383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.078902 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.078868 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.080356 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.080326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.086542 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.086508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.086827 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.086807 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkrg\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg\") pod \"e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.173917 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.173868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:18.328675 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.328652 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:53:18.333963 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:53:18.333927 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda03f813_c88c_4bb8_9cc3_e42abc90d848.slice/crio-d2519cd0c26b2ef5cf24e8f0297a115249fbbf8cd756ed373b61c358db122e10 WatchSource:0}: Error finding container d2519cd0c26b2ef5cf24e8f0297a115249fbbf8cd756ed373b61c358db122e10: Status 404 returned error can't find the container with id d2519cd0c26b2ef5cf24e8f0297a115249fbbf8cd756ed373b61c358db122e10 Jul 06 22:53:18.336999 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.336929 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:18.337083 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.337053 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:18.337120 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.337101 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:18.616681 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.616584 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerStarted","Data":"6e34a172c4b1dcc222baaad56565bc8d80bb12c6d09270652b46735130efdbe0"} Jul 06 22:53:18.616681 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.616651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerStarted","Data":"d2519cd0c26b2ef5cf24e8f0297a115249fbbf8cd756ed373b61c358db122e10"} Jul 06 22:53:18.634947 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:18.634894 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podStartSLOduration=1.63487217 podStartE2EDuration="1.63487217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:18.633182986 +0000 UTC m=+1754.659597353" watchObservedRunningTime="2026-07-06 22:53:18.63487217 +0000 UTC m=+1754.661286535" Jul 06 22:53:19.175098 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:19.175052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:19.180155 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:19.180127 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:19.621405 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:19.621326 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:19.622446 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:19.622428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:30.671185 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.671149 2576 generic.go:356] "Generic (PLEG): container finished" podID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerID="6e34a172c4b1dcc222baaad56565bc8d80bb12c6d09270652b46735130efdbe0" exitCode=0 Jul 06 22:53:30.671608 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.671219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerDied","Data":"6e34a172c4b1dcc222baaad56565bc8d80bb12c6d09270652b46735130efdbe0"} Jul 06 22:53:30.671608 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.671600 2576 scope.go:117] "RemoveContainer" containerID="6e34a172c4b1dcc222baaad56565bc8d80bb12c6d09270652b46735130efdbe0" Jul 06 22:53:30.672561 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.672531 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:30.672652 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.672616 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:30.672716 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:30.672689 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:53:31.680147 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:31.680113 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerStarted","Data":"ed8166aaaf7d20bec7ca6e67c61229d848a94a7bf9f5ce8c32ae07447f9e1ae2"} Jul 06 22:53:32.174349 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:32.174315 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:32.175616 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:32.175584 2576 patch_prober.go:28] interesting pod/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 06 22:53:32.175864 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:32.175832 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 06 22:53:33.174642 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:33.174595 2576 patch_prober.go:28] interesting pod/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 06 22:53:33.175186 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:33.174665 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 06 22:53:34.175314 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.175263 2576 patch_prober.go:28] interesting pod/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 06 22:53:34.175813 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.175349 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 06 22:53:34.281610 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.281577 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:53:34.285386 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.285360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.288130 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.288100 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75c546e0\"/\"openshift-service-ca.crt\"" Jul 06 22:53:34.289179 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.289146 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-75c546e0\"/\"kube-root-ca.crt\"" Jul 06 22:53:34.289373 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.289321 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75c546e0\"/\"default-dockercfg-jmrp2\"" Jul 06 22:53:34.289489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.289441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-75c546e0\"/\"e2e-lifecycle-model-75c546e0-kserve-self-signed-certs\"" Jul 06 22:53:34.291924 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.291902 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:53:34.423668 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423628 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.423668 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.423879 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.423879 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfkq\" (UniqueName: \"kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.423879 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.423879 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.423866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525240 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525150 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfkq\" (UniqueName: \"kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525240 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525240 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525240 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525578 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525578 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525899 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525787 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.525899 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.525814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.527504 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.527473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.527655 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.527637 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.532233 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.532216 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfkq\" (UniqueName: \"kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq\") pod \"e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.596498 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.596458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:34.621607 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.621576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:34.759003 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:34.758836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:53:34.780961 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:53:34.780930 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e79ed5a_4554_4512_8c17_6ab3c661daa5.slice/crio-d3cda40f0311512f85f2bca197a817fecef4ff4bab980ec19b039756c95ff3e6 WatchSource:0}: Error finding container d3cda40f0311512f85f2bca197a817fecef4ff4bab980ec19b039756c95ff3e6: Status 404 returned error can't find the container with id d3cda40f0311512f85f2bca197a817fecef4ff4bab980ec19b039756c95ff3e6 Jul 06 22:53:35.174996 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:35.174941 2576 patch_prober.go:28] interesting pod/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 06 22:53:35.175191 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:35.175041 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 06 22:53:35.699296 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:35.699260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" event={"ID":"4e79ed5a-4554-4512-8c17-6ab3c661daa5","Type":"ContainerStarted","Data":"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a"} Jul 06 22:53:35.699696 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:35.699302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" event={"ID":"4e79ed5a-4554-4512-8c17-6ab3c661daa5","Type":"ContainerStarted","Data":"d3cda40f0311512f85f2bca197a817fecef4ff4bab980ec19b039756c95ff3e6"} Jul 06 22:53:35.715258 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:35.715155 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" podStartSLOduration=1.715139471 podStartE2EDuration="1.715139471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:35.714942761 +0000 UTC m=+1771.741357125" watchObservedRunningTime="2026-07-06 22:53:35.715139471 +0000 UTC m=+1771.741553834" Jul 06 22:53:36.178138 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:36.178109 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:36.179098 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:36.179080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:53:44.597424 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:44.597385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:44.597807 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:44.597437 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:44.610287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:44.610260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:53:44.746725 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:53:44.746697 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:54:18.864278 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:18.864226 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:54:33.885705 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:33.885640 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" podUID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" containerName="main" containerID="cri-o://44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a" gracePeriod=45 Jul 06 22:54:34.132260 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.132235 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:54:34.241588 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241470 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtfkq\" (UniqueName: \"kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241588 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241546 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241588 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241571 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241588 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241628 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache\") pod \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\" (UID: \"4e79ed5a-4554-4512-8c17-6ab3c661daa5\") " Jul 06 22:54:34.241934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241877 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home" (OuterVolumeSpecName: "home") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:54:34.241934 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.241887 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:54:34.242180 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.242003 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:54:34.243850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.243826 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm" (OuterVolumeSpecName: "dshm") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:54:34.243850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.243831 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:54:34.243969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.243944 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq" (OuterVolumeSpecName: "kube-api-access-qtfkq") pod "4e79ed5a-4554-4512-8c17-6ab3c661daa5" (UID: "4e79ed5a-4554-4512-8c17-6ab3c661daa5"). InnerVolumeSpecName "kube-api-access-qtfkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:54:34.342964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.342925 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.342964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.342956 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.342964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.342965 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfkq\" (UniqueName: \"kubernetes.io/projected/4e79ed5a-4554-4512-8c17-6ab3c661daa5-kube-api-access-qtfkq\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.342964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.342985 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.342964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.342995 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.343285 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.343003 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e79ed5a-4554-4512-8c17-6ab3c661daa5-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:54:34.945047 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.945008 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" containerID="44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a" exitCode=0 Jul 06 22:54:34.945499 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.945055 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" Jul 06 22:54:34.945499 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.945093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" event={"ID":"4e79ed5a-4554-4512-8c17-6ab3c661daa5","Type":"ContainerDied","Data":"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a"} Jul 06 22:54:34.945499 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.945137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb" event={"ID":"4e79ed5a-4554-4512-8c17-6ab3c661daa5","Type":"ContainerDied","Data":"d3cda40f0311512f85f2bca197a817fecef4ff4bab980ec19b039756c95ff3e6"} Jul 06 22:54:34.945499 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.945158 2576 scope.go:117] "RemoveContainer" containerID="44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a" Jul 06 22:54:34.953414 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.953396 2576 scope.go:117] "RemoveContainer" containerID="44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a" Jul 06 22:54:34.953701 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:54:34.953682 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a\": container with ID starting with 44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a not found: ID does not exist" containerID="44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a" Jul 06 22:54:34.953781 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.953711 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a"} err="failed to get container status \"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a\": rpc error: code = NotFound desc = could not find container \"44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a\": container with ID starting with 44f143591f37cc1e2db23677d544105ac9223b8b42840dd9d4005b2bfb32826a not found: ID does not exist" Jul 06 22:54:34.962023 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.961990 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:54:34.966669 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:34.966633 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-75c546e0/e2e-lifecycle-model-75c546e0-kserve-594ff46459-grsnb"] Jul 06 22:54:36.520957 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:54:36.520923 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" path="/var/lib/kubelet/pods/4e79ed5a-4554-4512-8c17-6ab3c661daa5/volumes" Jul 06 22:55:02.905741 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:02.905704 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:55:02.906188 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:02.905968 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" containerID="cri-o://ed8166aaaf7d20bec7ca6e67c61229d848a94a7bf9f5ce8c32ae07447f9e1ae2" gracePeriod=30 Jul 06 22:55:06.180838 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:06.180787 2576 patch_prober.go:28] interesting pod/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 22:55:06.181322 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:06.180862 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 22:55:08.094290 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.094252 2576 generic.go:356] "Generic (PLEG): container finished" podID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerID="ed8166aaaf7d20bec7ca6e67c61229d848a94a7bf9f5ce8c32ae07447f9e1ae2" exitCode=0 Jul 06 22:55:08.094745 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.094319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerDied","Data":"ed8166aaaf7d20bec7ca6e67c61229d848a94a7bf9f5ce8c32ae07447f9e1ae2"} Jul 06 22:55:08.094745 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.094386 2576 scope.go:117] "RemoveContainer" containerID="6e34a172c4b1dcc222baaad56565bc8d80bb12c6d09270652b46735130efdbe0" Jul 06 22:55:08.167622 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.167554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:55:08.235255 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235255 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235258 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235336 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235372 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235397 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235420 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngkrg\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235459 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235483 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert\") pod \"da03f813-c88c-4bb8-9cc3-e42abc90d848\" (UID: \"da03f813-c88c-4bb8-9cc3-e42abc90d848\") " Jul 06 22:55:08.235513 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235501 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:08.235915 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235607 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:08.235915 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235719 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.235915 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235738 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.235915 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235733 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:08.236159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.235914 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:55:08.236306 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.236283 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data" (OuterVolumeSpecName: "istio-data") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:08.237839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.237808 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:55:08.237839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.237807 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:08.238122 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.238097 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg" (OuterVolumeSpecName: "kube-api-access-ngkrg") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "kube-api-access-ngkrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:55:08.238190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.238130 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token" (OuterVolumeSpecName: "istio-token") pod "da03f813-c88c-4bb8-9cc3-e42abc90d848" (UID: "da03f813-c88c-4bb8-9cc3-e42abc90d848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:55:08.336964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.336929 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.336964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.336957 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da03f813-c88c-4bb8-9cc3-e42abc90d848-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.336964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.336965 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.336964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.336991 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.336964 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.337000 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.337283 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.337009 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da03f813-c88c-4bb8-9cc3-e42abc90d848-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:08.337283 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:08.337018 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkrg\" (UniqueName: \"kubernetes.io/projected/da03f813-c88c-4bb8-9cc3-e42abc90d848-kube-api-access-ngkrg\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:55:09.100565 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:09.100525 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" event={"ID":"da03f813-c88c-4bb8-9cc3-e42abc90d848","Type":"ContainerDied","Data":"d2519cd0c26b2ef5cf24e8f0297a115249fbbf8cd756ed373b61c358db122e10"} Jul 06 22:55:09.101075 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:09.100564 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp" Jul 06 22:55:09.101075 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:09.100602 2576 scope.go:117] "RemoveContainer" containerID="ed8166aaaf7d20bec7ca6e67c61229d848a94a7bf9f5ce8c32ae07447f9e1ae2" Jul 06 22:55:09.120024 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:09.119965 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:55:09.124755 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:09.124719 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-75c546e0-openshift-default-7d9c48f599-g4kpp"] Jul 06 22:55:10.521778 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:10.521740 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" path="/var/lib/kubelet/pods/da03f813-c88c-4bb8-9cc3-e42abc90d848/volumes" Jul 06 22:55:58.505935 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.505901 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506206 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" containerName="main" Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506217 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" containerName="main" Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506226 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506232 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506290 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e79ed5a-4554-4512-8c17-6ab3c661daa5" containerName="main" Jul 06 22:55:58.506395 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.506298 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:55:58.509310 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.509290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.512250 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.512226 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8f9c60-openshift-default-dockercfg-f964z\"" Jul 06 22:55:58.522085 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.522064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:55:58.557946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.557912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.557946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.557946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.558177 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.557965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.558177 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.558063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.558177 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.558104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.564189 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.558287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.564412 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.564391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.564457 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.564434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.564491 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.564462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc6f\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665533 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc6f\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.665795 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.665787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.666161 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.666140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.666264 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.666244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.666379 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.666353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.666491 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.666383 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.666491 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.666451 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.668320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.668298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.668449 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.668432 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.673438 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.673409 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc6f\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.673557 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.673461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token\") pod \"e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.819941 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.819840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:55:58.967363 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.967324 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:55:58.970506 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:55:58.970468 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13269599_8095_47cd_8050_7ea2d8f3ad11.slice/crio-d0f4fddb4c605d282e893ed5bd966564acd41dff340e5526d6de595df9892a92 WatchSource:0}: Error finding container d0f4fddb4c605d282e893ed5bd966564acd41dff340e5526d6de595df9892a92: Status 404 returned error can't find the container with id d0f4fddb4c605d282e893ed5bd966564acd41dff340e5526d6de595df9892a92 Jul 06 22:55:58.972662 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.972620 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:55:58.972764 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.972702 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:55:58.972764 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:58.972732 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:55:59.311483 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:59.311446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerStarted","Data":"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b"} Jul 06 22:55:59.311643 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:59.311504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerStarted","Data":"d0f4fddb4c605d282e893ed5bd966564acd41dff340e5526d6de595df9892a92"} Jul 06 22:55:59.330845 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:59.330794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podStartSLOduration=1.330772879 podStartE2EDuration="1.330772879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:59.330472782 +0000 UTC m=+1915.356887159" watchObservedRunningTime="2026-07-06 22:55:59.330772879 +0000 UTC m=+1915.357187246" Jul 06 22:55:59.820663 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:55:59.820613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:00.826086 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:00.826057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:01.322526 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:01.322495 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:01.323660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:01.323641 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:16.324154 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.324115 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:56:16.324578 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.324190 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:56:16.394449 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.394418 2576 generic.go:356] "Generic (PLEG): container finished" podID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerID="65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b" exitCode=0 Jul 06 22:56:16.394621 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.394461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerDied","Data":"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b"} Jul 06 22:56:16.394830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.394813 2576 scope.go:117] "RemoveContainer" containerID="65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b" Jul 06 22:56:16.395830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.395792 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:56:16.395926 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.395896 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:56:16.396015 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:16.395953 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:56:17.401456 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:17.401416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerStarted","Data":"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72"} Jul 06 22:56:17.820435 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:17.820342 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:17.820699 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:17.820630 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:56:17.820777 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:17.820730 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:56:18.821268 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:18.821229 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:56:18.821676 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:18.821286 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:56:19.820532 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:19.820496 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:56:19.820714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:19.820549 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:56:20.569511 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.569468 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:56:20.570065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.570040 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:56:20.570174 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.570067 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:56:20.570245 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.570186 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="da03f813-c88c-4bb8-9cc3-e42abc90d848" containerName="istio-proxy" Jul 06 22:56:20.572821 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.572796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.575489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.575462 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8f9c60\"/\"kube-root-ca.crt\"" Jul 06 22:56:20.576702 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.576673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8f9c60\"/\"test-model-8f9c60-kserve-self-signed-certs\"" Jul 06 22:56:20.576998 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.576959 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8f9c60\"/\"openshift-service-ca.crt\"" Jul 06 22:56:20.577223 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.577201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8f9c60\"/\"default-dockercfg-wskcw\"" Jul 06 22:56:20.590575 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.590547 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnh9\" (UniqueName: \"kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.671002 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.664639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.765852 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.765806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767018 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.766994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnh9\" (UniqueName: \"kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767240 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.767220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767381 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.767366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767507 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.767493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767683 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.767668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.767922 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.767867 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.768287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.768243 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.768597 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.768579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.771184 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.771163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.776998 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.772570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.776998 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.776627 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnh9\" (UniqueName: \"kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9\") pod \"test-model-8f9c60-kserve-8cb9d4cd9-94pzz\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:20.820614 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.820521 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:56:20.820614 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.820578 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:56:20.886742 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:20.886702 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:21.043706 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.043680 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:56:21.046406 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:56:21.046374 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64fb2c24_5409_43cc_aba9_0d22be69d97a.slice/crio-cd4372d21c75d5766f9acc450c832bf8ab5678afb6aa71cd6325750f591dfb8c WatchSource:0}: Error finding container cd4372d21c75d5766f9acc450c832bf8ab5678afb6aa71cd6325750f591dfb8c: Status 404 returned error can't find the container with id cd4372d21c75d5766f9acc450c832bf8ab5678afb6aa71cd6325750f591dfb8c Jul 06 22:56:21.420220 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.420187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" event={"ID":"64fb2c24-5409-43cc-aba9-0d22be69d97a","Type":"ContainerStarted","Data":"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228"} Jul 06 22:56:21.420392 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.420228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" event={"ID":"64fb2c24-5409-43cc-aba9-0d22be69d97a","Type":"ContainerStarted","Data":"cd4372d21c75d5766f9acc450c832bf8ab5678afb6aa71cd6325750f591dfb8c"} Jul 06 22:56:21.437366 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.437317 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" podStartSLOduration=1.437301344 podStartE2EDuration="1.437301344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:21.435677332 +0000 UTC m=+1937.462091693" watchObservedRunningTime="2026-07-06 22:56:21.437301344 +0000 UTC m=+1937.463715707" Jul 06 22:56:21.824725 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.824643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:21.825127 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.824930 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:21.825678 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:21.825660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:56:30.887272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:30.887227 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:30.887667 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:30.887291 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:30.899932 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:30.899902 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:31.475652 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:31.475623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:56:57.681309 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.681229 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:56:57.684753 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.684734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.688805 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.688777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-878700\"/\"default-dockercfg-lvm22\"" Jul 06 22:56:57.689229 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.689209 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-878700\"/\"openshift-service-ca.crt\"" Jul 06 22:56:57.690381 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.690356 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-878700\"/\"kube-root-ca.crt\"" Jul 06 22:56:57.691185 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.691162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-878700\"/\"test-model-878700-kserve-self-signed-certs\"" Jul 06 22:56:57.698393 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.698361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:56:57.790874 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.790828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.791089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.790882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgjp2\" (UniqueName: \"kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.791089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.790941 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.791089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.790968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.791089 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.791058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.791263 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.791114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892531 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892531 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892789 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892789 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgjp2\" (UniqueName: \"kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892789 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.892789 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.893053 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.892971 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.893124 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.893071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.893124 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.893100 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.894957 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.894938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.895244 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.895228 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.901263 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.901244 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgjp2\" (UniqueName: \"kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2\") pod \"test-model-878700-kserve-85d876748f-b7vzh\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:57.995937 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:57.995849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:56:58.127488 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:58.127466 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:56:58.130028 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:56:58.129996 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d880490_addc_4e00_97ce_037d00efaece.slice/crio-a854116c06fb56740db9cc5601b982eabfbdf094efb0ebc0108813c95db29e5b WatchSource:0}: Error finding container a854116c06fb56740db9cc5601b982eabfbdf094efb0ebc0108813c95db29e5b: Status 404 returned error can't find the container with id a854116c06fb56740db9cc5601b982eabfbdf094efb0ebc0108813c95db29e5b Jul 06 22:56:58.578196 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:58.578160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" event={"ID":"2d880490-addc-4e00-97ce-037d00efaece","Type":"ContainerStarted","Data":"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd"} Jul 06 22:56:58.578196 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:58.578200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" event={"ID":"2d880490-addc-4e00-97ce-037d00efaece","Type":"ContainerStarted","Data":"a854116c06fb56740db9cc5601b982eabfbdf094efb0ebc0108813c95db29e5b"} Jul 06 22:56:58.594598 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:56:58.594531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" podStartSLOduration=1.594511727 podStartE2EDuration="1.594511727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:58.593963168 +0000 UTC m=+1974.620377532" watchObservedRunningTime="2026-07-06 22:56:58.594511727 +0000 UTC m=+1974.620926092" Jul 06 22:57:07.996272 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:07.996235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:07.996749 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:07.996294 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:08.008814 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:08.008787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:08.634539 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:08.634507 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:32.783646 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:32.783602 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:57:33.777877 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:33.777844 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:57:47.804569 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:47.804529 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" podUID="64fb2c24-5409-43cc-aba9-0d22be69d97a" containerName="main" containerID="cri-o://438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228" gracePeriod=45 Jul 06 22:57:48.052020 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.051995 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:57:48.118269 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118176 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118269 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118269 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118562 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118301 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118562 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118321 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtnh9\" (UniqueName: \"kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118562 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118347 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs\") pod \"64fb2c24-5409-43cc-aba9-0d22be69d97a\" (UID: \"64fb2c24-5409-43cc-aba9-0d22be69d97a\") " Jul 06 22:57:48.118562 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118519 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home" (OuterVolumeSpecName: "home") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:48.118710 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118574 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache" (OuterVolumeSpecName: "model-cache") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:48.118710 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.118592 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:48.120598 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.120564 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:57:48.120598 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.120569 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9" (OuterVolumeSpecName: "kube-api-access-gtnh9") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "kube-api-access-gtnh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:57:48.120737 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.120634 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm" (OuterVolumeSpecName: "dshm") pod "64fb2c24-5409-43cc-aba9-0d22be69d97a" (UID: "64fb2c24-5409-43cc-aba9-0d22be69d97a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:48.219210 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219169 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.219210 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219199 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnh9\" (UniqueName: \"kubernetes.io/projected/64fb2c24-5409-43cc-aba9-0d22be69d97a-kube-api-access-gtnh9\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.219210 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219212 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64fb2c24-5409-43cc-aba9-0d22be69d97a-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.219210 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219220 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.219482 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219228 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.219482 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.219236 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64fb2c24-5409-43cc-aba9-0d22be69d97a-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:48.794727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.794633 2576 generic.go:356] "Generic (PLEG): container finished" podID="64fb2c24-5409-43cc-aba9-0d22be69d97a" containerID="438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228" exitCode=0 Jul 06 22:57:48.794727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.794685 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" Jul 06 22:57:48.794727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.794690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" event={"ID":"64fb2c24-5409-43cc-aba9-0d22be69d97a","Type":"ContainerDied","Data":"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228"} Jul 06 22:57:48.794727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.794729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz" event={"ID":"64fb2c24-5409-43cc-aba9-0d22be69d97a","Type":"ContainerDied","Data":"cd4372d21c75d5766f9acc450c832bf8ab5678afb6aa71cd6325750f591dfb8c"} Jul 06 22:57:48.795049 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.794751 2576 scope.go:117] "RemoveContainer" containerID="438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228" Jul 06 22:57:48.798401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.798360 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" podUID="2d880490-addc-4e00-97ce-037d00efaece" containerName="main" containerID="cri-o://486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd" gracePeriod=45 Jul 06 22:57:48.803660 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.803637 2576 scope.go:117] "RemoveContainer" containerID="438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228" Jul 06 22:57:48.803989 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:57:48.803958 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228\": container with ID starting with 438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228 not found: ID does not exist" containerID="438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228" Jul 06 22:57:48.804056 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.804004 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228"} err="failed to get container status \"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228\": rpc error: code = NotFound desc = could not find container \"438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228\": container with ID starting with 438c22dade001602ae2f8d3af5c798c377156f015d6bca0c8e3a8de3fccc5228 not found: ID does not exist" Jul 06 22:57:48.812183 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.812155 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:57:48.814534 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:48.814506 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8f9c60/test-model-8f9c60-kserve-8cb9d4cd9-94pzz"] Jul 06 22:57:49.054777 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.054704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:49.125739 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.125912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.125912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125787 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.125912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125808 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgjp2\" (UniqueName: \"kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.125912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.125912 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.125850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs\") pod \"2d880490-addc-4e00-97ce-037d00efaece\" (UID: \"2d880490-addc-4e00-97ce-037d00efaece\") " Jul 06 22:57:49.126203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.126135 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:49.126203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.126149 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:49.126203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.126168 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home" (OuterVolumeSpecName: "home") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:49.128121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.128094 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2" (OuterVolumeSpecName: "kube-api-access-rgjp2") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "kube-api-access-rgjp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:57:49.128121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.128106 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm" (OuterVolumeSpecName: "dshm") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:57:49.128121 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.128115 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d880490-addc-4e00-97ce-037d00efaece" (UID: "2d880490-addc-4e00-97ce-037d00efaece"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:57:49.227254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227204 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.227254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227247 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.227254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227257 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgjp2\" (UniqueName: \"kubernetes.io/projected/2d880490-addc-4e00-97ce-037d00efaece-kube-api-access-rgjp2\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.227254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227267 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.227254 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227276 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d880490-addc-4e00-97ce-037d00efaece-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.227560 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.227286 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d880490-addc-4e00-97ce-037d00efaece-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:57:49.805525 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.805482 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d880490-addc-4e00-97ce-037d00efaece" containerID="486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd" exitCode=0 Jul 06 22:57:49.805701 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.805535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" event={"ID":"2d880490-addc-4e00-97ce-037d00efaece","Type":"ContainerDied","Data":"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd"} Jul 06 22:57:49.805701 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.805541 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" Jul 06 22:57:49.805701 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.805568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh" event={"ID":"2d880490-addc-4e00-97ce-037d00efaece","Type":"ContainerDied","Data":"a854116c06fb56740db9cc5601b982eabfbdf094efb0ebc0108813c95db29e5b"} Jul 06 22:57:49.805701 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.805587 2576 scope.go:117] "RemoveContainer" containerID="486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd" Jul 06 22:57:49.815837 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.815810 2576 scope.go:117] "RemoveContainer" containerID="486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd" Jul 06 22:57:49.816206 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:57:49.816186 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd\": container with ID starting with 486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd not found: ID does not exist" containerID="486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd" Jul 06 22:57:49.816259 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.816215 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd"} err="failed to get container status \"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd\": rpc error: code = NotFound desc = could not find container \"486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd\": container with ID starting with 486cbcacae86bb854e9ccbf85b0277f01f303074da1f04cf62009bfc02b543bd not found: ID does not exist" Jul 06 22:57:49.827460 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.827421 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:57:49.829023 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:49.828998 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-878700/test-model-878700-kserve-85d876748f-b7vzh"] Jul 06 22:57:50.521958 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:50.521925 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d880490-addc-4e00-97ce-037d00efaece" path="/var/lib/kubelet/pods/2d880490-addc-4e00-97ce-037d00efaece/volumes" Jul 06 22:57:50.522350 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:57:50.522336 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fb2c24-5409-43cc-aba9-0d22be69d97a" path="/var/lib/kubelet/pods/64fb2c24-5409-43cc-aba9-0d22be69d97a/volumes" Jul 06 22:58:16.668294 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:16.668255 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:58:16.668793 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:16.668515 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" containerID="cri-o://4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72" gracePeriod=30 Jul 06 22:58:21.826452 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.826420 2576 patch_prober.go:28] interesting pod/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:58:21.826862 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.826487 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 06 22:58:21.919910 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.919886 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:58:21.942597 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.942516 2576 generic.go:356] "Generic (PLEG): container finished" podID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerID="4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72" exitCode=0 Jul 06 22:58:21.942597 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.942564 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" Jul 06 22:58:21.942776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.942603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerDied","Data":"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72"} Jul 06 22:58:21.942776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.942655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn" event={"ID":"13269599-8095-47cd-8050-7ea2d8f3ad11","Type":"ContainerDied","Data":"d0f4fddb4c605d282e893ed5bd966564acd41dff340e5526d6de595df9892a92"} Jul 06 22:58:21.942776 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.942676 2576 scope.go:117] "RemoveContainer" containerID="4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72" Jul 06 22:58:21.951714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.951691 2576 scope.go:117] "RemoveContainer" containerID="65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b" Jul 06 22:58:21.959287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.959270 2576 scope.go:117] "RemoveContainer" containerID="4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72" Jul 06 22:58:21.959562 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:58:21.959543 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72\": container with ID starting with 4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72 not found: ID does not exist" containerID="4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72" Jul 06 22:58:21.959633 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.959574 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72"} err="failed to get container status \"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72\": rpc error: code = NotFound desc = could not find container \"4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72\": container with ID starting with 4f9fc3d8729523dd527ada32e6b8d9559a63940e339590f9c586da0b5f6baa72 not found: ID does not exist" Jul 06 22:58:21.959633 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.959598 2576 scope.go:117] "RemoveContainer" containerID="65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b" Jul 06 22:58:21.959859 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:58:21.959833 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b\": container with ID starting with 65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b not found: ID does not exist" containerID="65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b" Jul 06 22:58:21.959904 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:21.959871 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b"} err="failed to get container status \"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b\": rpc error: code = NotFound desc = could not find container \"65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b\": container with ID starting with 65eca6757f26f82547e23841e370a159857a5af106f5fec972fa0adc8727307b not found: ID does not exist" Jul 06 22:58:22.002348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002317 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002363 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002400 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjc6f\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002486 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002695 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002521 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002695 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002695 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002579 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002695 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002606 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token\") pod \"13269599-8095-47cd-8050-7ea2d8f3ad11\" (UID: \"13269599-8095-47cd-8050-7ea2d8f3ad11\") " Jul 06 22:58:22.002902 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002697 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:58:22.002962 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002883 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:58:22.003050 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.002957 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.003159 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.003129 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:58:22.003234 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.003178 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:58:22.003852 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.003824 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data" (OuterVolumeSpecName: "istio-data") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:58:22.004828 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.004677 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f" (OuterVolumeSpecName: "kube-api-access-vjc6f") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "kube-api-access-vjc6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:58:22.005189 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.005148 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token" (OuterVolumeSpecName: "istio-token") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:58:22.005285 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.005165 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:58:22.005656 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.005389 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13269599-8095-47cd-8050-7ea2d8f3ad11" (UID: "13269599-8095-47cd-8050-7ea2d8f3ad11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:58:22.104080 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104038 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjc6f\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-kube-api-access-vjc6f\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104080 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104072 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13269599-8095-47cd-8050-7ea2d8f3ad11-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104080 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104087 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104100 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104109 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104117 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104124 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.104320 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.104131 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13269599-8095-47cd-8050-7ea2d8f3ad11-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:58:22.264428 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.264399 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:58:22.267969 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.267944 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8f9c60-openshift-default-6f848dc7c7-vlnfn"] Jul 06 22:58:22.521090 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:22.521011 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" path="/var/lib/kubelet/pods/13269599-8095-47cd-8050-7ea2d8f3ad11/volumes" Jul 06 22:58:24.591104 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591072 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591378 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fb2c24-5409-43cc-aba9-0d22be69d97a" containerName="main" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591390 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fb2c24-5409-43cc-aba9-0d22be69d97a" containerName="main" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591399 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d880490-addc-4e00-97ce-037d00efaece" containerName="main" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591405 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d880490-addc-4e00-97ce-037d00efaece" containerName="main" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591427 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591436 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591447 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591451 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591453 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591506 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591515 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fb2c24-5409-43cc-aba9-0d22be69d97a" containerName="main" Jul 06 22:58:24.591727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591520 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13269599-8095-47cd-8050-7ea2d8f3ad11" containerName="istio-proxy" Jul 06 22:58:24.591727 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.591526 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d880490-addc-4e00-97ce-037d00efaece" containerName="main" Jul 06 22:58:24.595592 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.595573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.598287 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.598269 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8090dc-openshift-default-dockercfg-zpk45\"" Jul 06 22:58:24.610946 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.610923 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:58:24.728940 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.728904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729155 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.728949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729155 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.728972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729155 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.729327 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.729311 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwtkm\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830112 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830112 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830231 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830340 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830301 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830493 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwtkm\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830493 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830644 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830703 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.830963 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.830943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.831056 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.831036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.831103 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.831088 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.832817 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.832796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.833142 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.833122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.839060 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.839035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.839186 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.839166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwtkm\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm\") pod \"e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:24.907126 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:24.907095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:25.060572 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.060546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:58:25.062051 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:58:25.062019 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda71b29e_3b7b_4e83_a8db_b20d1594fcaf.slice/crio-e3064c1d6634927c3bcbbd84edb9770117f412005622fe42829fdbc49c056db2 WatchSource:0}: Error finding container e3064c1d6634927c3bcbbd84edb9770117f412005622fe42829fdbc49c056db2: Status 404 returned error can't find the container with id e3064c1d6634927c3bcbbd84edb9770117f412005622fe42829fdbc49c056db2 Jul 06 22:58:25.064114 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.064079 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:25.064213 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.064149 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:25.064213 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.064177 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:25.960932 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.960892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerStarted","Data":"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d"} Jul 06 22:58:25.961348 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.960951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerStarted","Data":"e3064c1d6634927c3bcbbd84edb9770117f412005622fe42829fdbc49c056db2"} Jul 06 22:58:25.981065 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:25.981017 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podStartSLOduration=1.981001505 podStartE2EDuration="1.981001505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:58:25.979452414 +0000 UTC m=+2062.005866778" watchObservedRunningTime="2026-07-06 22:58:25.981001505 +0000 UTC m=+2062.007415859" Jul 06 22:58:26.907998 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:26.907944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:26.912932 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:26.912909 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:26.966133 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:26.966102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:26.967073 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:26.967057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:41.968073 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:41.968036 2576 patch_prober.go:28] interesting pod/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 22:58:41.968492 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:41.968100 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 22:58:42.027920 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.027880 2576 generic.go:356] "Generic (PLEG): container finished" podID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerID="2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d" exitCode=0 Jul 06 22:58:42.028125 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.027956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerDied","Data":"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d"} Jul 06 22:58:42.028401 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.028382 2576 scope.go:117] "RemoveContainer" containerID="2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d" Jul 06 22:58:42.029425 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.029392 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:42.029529 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.029484 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:42.029597 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:42.029544 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:58:43.035337 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:43.035293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerStarted","Data":"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab"} Jul 06 22:58:43.908307 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:43.908262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:43.908548 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:43.908523 2576 patch_prober.go:28] interesting pod/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 22:58:43.908631 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:43.908580 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 22:58:44.907832 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:44.907794 2576 patch_prober.go:28] interesting pod/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 22:58:44.908262 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:44.907851 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 22:58:45.908517 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:45.908480 2576 patch_prober.go:28] interesting pod/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 22:58:45.908948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:45.908541 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 22:58:46.908034 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:46.907992 2576 patch_prober.go:28] interesting pod/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 06 22:58:46.908203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:46.908070 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 06 22:58:47.911822 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:47.911789 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:47.912228 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:47.912017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:47.912842 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:47.912821 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:58:55.652238 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:55.652200 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:58:55.652685 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:58:55.652419 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" containerID="cri-o://673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab" gracePeriod=30 Jul 06 22:59:00.897696 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.897666 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:59:00.933822 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933745 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.933822 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.933822 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933808 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933831 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933846 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933913 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwtkm\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933947 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.933972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934019 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data\") pod \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\" (UID: \"da71b29e-3b7b-4e83-a8db-b20d1594fcaf\") " Jul 06 22:59:00.934257 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934118 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:00.934729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934275 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:00.934729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934273 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:59:00.934729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934368 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:00.934729 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.934437 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:00.935311 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.935280 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data" (OuterVolumeSpecName: "istio-data") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:00.936283 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.936244 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:59:00.936362 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.936277 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token" (OuterVolumeSpecName: "istio-token") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:59:00.936362 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.936331 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:59:00.936545 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:00.936518 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm" (OuterVolumeSpecName: "kube-api-access-rwtkm") pod "da71b29e-3b7b-4e83-a8db-b20d1594fcaf" (UID: "da71b29e-3b7b-4e83-a8db-b20d1594fcaf"). InnerVolumeSpecName "kube-api-access-rwtkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:59:01.035213 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035174 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035213 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035207 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035213 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035217 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035225 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035240 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035247 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035255 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.035447 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.035264 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwtkm\" (UniqueName: \"kubernetes.io/projected/da71b29e-3b7b-4e83-a8db-b20d1594fcaf-kube-api-access-rwtkm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 22:59:01.114972 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.114936 2576 generic.go:356] "Generic (PLEG): container finished" podID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerID="673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab" exitCode=0 Jul 06 22:59:01.115190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.115000 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" Jul 06 22:59:01.115190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.115024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerDied","Data":"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab"} Jul 06 22:59:01.115190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.115089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5" event={"ID":"da71b29e-3b7b-4e83-a8db-b20d1594fcaf","Type":"ContainerDied","Data":"e3064c1d6634927c3bcbbd84edb9770117f412005622fe42829fdbc49c056db2"} Jul 06 22:59:01.115190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.115119 2576 scope.go:117] "RemoveContainer" containerID="673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab" Jul 06 22:59:01.124195 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.124175 2576 scope.go:117] "RemoveContainer" containerID="2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d" Jul 06 22:59:01.132126 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.132107 2576 scope.go:117] "RemoveContainer" containerID="673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab" Jul 06 22:59:01.132399 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:59:01.132379 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab\": container with ID starting with 673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab not found: ID does not exist" containerID="673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab" Jul 06 22:59:01.132450 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.132407 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab"} err="failed to get container status \"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab\": rpc error: code = NotFound desc = could not find container \"673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab\": container with ID starting with 673591b0a5b348520d2c982046532110119ca6372d9f3c7eb1c94a16a493c8ab not found: ID does not exist" Jul 06 22:59:01.132450 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.132427 2576 scope.go:117] "RemoveContainer" containerID="2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d" Jul 06 22:59:01.132651 ip-10-0-139-147 kubenswrapper[2576]: E0706 22:59:01.132633 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d\": container with ID starting with 2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d not found: ID does not exist" containerID="2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d" Jul 06 22:59:01.132711 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.132661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d"} err="failed to get container status \"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d\": rpc error: code = NotFound desc = could not find container \"2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d\": container with ID starting with 2d87fd28420f7e4f38719be0ef7d5e02b2964419f329152a0de9841acafc004d not found: ID does not exist" Jul 06 22:59:01.137342 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.137317 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:59:01.144959 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:01.144934 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8090dc-openshift-default-7d85dbff99-gc8d5"] Jul 06 22:59:02.521052 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:02.521016 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" path="/var/lib/kubelet/pods/da71b29e-3b7b-4e83-a8db-b20d1594fcaf/volumes" Jul 06 22:59:03.972850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.972817 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 22:59:03.973293 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.973136 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:03.973293 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.973147 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:03.973293 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.973197 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:03.973293 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.973207 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:03.975921 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.975904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:03.978314 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.978294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c8326f-openshift-default-dockercfg-7v2wf\"" Jul 06 22:59:03.990560 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:03.990530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 22:59:04.063466 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063645 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.063785 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.063773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvhfd\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165012 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.164961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165012 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165292 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165565 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165565 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvhfd\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165565 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165565 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165530 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165827 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165575 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.165827 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.165692 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.166046 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.166028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.167649 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.167628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.167774 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.167755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.173202 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.173173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvhfd\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.173413 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.173392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token\") pod \"e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.285949 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.285851 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:04.430850 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.430821 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 22:59:04.432807 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:59:04.432769 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1fe7c6_994a_4764_ac30_82e625c09e66.slice/crio-d7c63bf65a14e10697ae02ea4d377dbecff59cfd96588f4345bca6290f06ac5f WatchSource:0}: Error finding container d7c63bf65a14e10697ae02ea4d377dbecff59cfd96588f4345bca6290f06ac5f: Status 404 returned error can't find the container with id d7c63bf65a14e10697ae02ea4d377dbecff59cfd96588f4345bca6290f06ac5f Jul 06 22:59:04.435306 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.435269 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:04.435408 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.435349 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:04.435408 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:04.435397 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:05.134952 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:05.134912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerStarted","Data":"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b"} Jul 06 22:59:05.135381 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:05.135002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerStarted","Data":"d7c63bf65a14e10697ae02ea4d377dbecff59cfd96588f4345bca6290f06ac5f"} Jul 06 22:59:05.155289 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:05.155239 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podStartSLOduration=2.15522029 podStartE2EDuration="2.15522029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:59:05.153793939 +0000 UTC m=+2101.180208305" watchObservedRunningTime="2026-07-06 22:59:05.15522029 +0000 UTC m=+2101.181634655" Jul 06 22:59:05.286972 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:05.286934 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:05.291739 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:05.291706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:06.139520 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:06.139483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:06.140616 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:06.140598 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:12.141262 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.141218 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 22:59:12.141890 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.141682 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:12.141890 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.141697 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="da71b29e-3b7b-4e83-a8db-b20d1594fcaf" containerName="istio-proxy" Jul 06 22:59:12.144771 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.144749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.147620 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.147600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b0de62-openshift-default-dockercfg-7tmvp\"" Jul 06 22:59:12.154836 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.154806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 22:59:12.235019 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.234965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x792\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235203 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235159 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235334 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235334 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235334 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235334 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.235487 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.235345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336359 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336566 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x792\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.336830 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.336792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.337180 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.337162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.337258 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.337199 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.337258 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.337246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.337594 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.337578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.338965 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.338947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.339273 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.339255 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.343709 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.343688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x792\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.343778 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.343751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token\") pod \"e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.456173 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.456080 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:12.599768 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.599743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 22:59:12.602345 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:59:12.602316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd931979e_beaf_45a2_adad_c57afd9ebfee.slice/crio-bbc19156be3dbc74c539c90a2330f7af423f2190eff15f07182b484ac4b995a1 WatchSource:0}: Error finding container bbc19156be3dbc74c539c90a2330f7af423f2190eff15f07182b484ac4b995a1: Status 404 returned error can't find the container with id bbc19156be3dbc74c539c90a2330f7af423f2190eff15f07182b484ac4b995a1 Jul 06 22:59:12.604516 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.604480 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:12.604604 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.604552 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:12.604604 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:12.604582 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:13.174773 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:13.174724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerStarted","Data":"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4"} Jul 06 22:59:13.175306 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:13.174814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerStarted","Data":"bbc19156be3dbc74c539c90a2330f7af423f2190eff15f07182b484ac4b995a1"} Jul 06 22:59:13.197178 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:13.197125 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podStartSLOduration=1.197105935 podStartE2EDuration="1.197105935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:59:13.195466087 +0000 UTC m=+2109.221880470" watchObservedRunningTime="2026-07-06 22:59:13.197105935 +0000 UTC m=+2109.223520299" Jul 06 22:59:13.456661 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:13.456559 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:13.467017 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:13.466968 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:14.179721 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:14.179691 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:14.180754 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:14.180735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:20.209271 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.209235 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerID="4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b" exitCode=0 Jul 06 22:59:20.209714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.209309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerDied","Data":"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b"} Jul 06 22:59:20.209714 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.209712 2576 scope.go:117] "RemoveContainer" containerID="4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b" Jul 06 22:59:20.210754 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.210725 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:20.210832 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.210814 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:20.210880 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:20.210864 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:21.140169 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:21.140131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:21.215103 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:21.215067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerStarted","Data":"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35"} Jul 06 22:59:21.286793 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:21.286748 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:21.287948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:21.287924 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 06 22:59:21.288073 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:21.287971 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 06 22:59:22.286617 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:22.286583 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 06 22:59:22.287136 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:22.286640 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 06 22:59:23.287465 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:23.287417 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 06 22:59:23.288039 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:23.288004 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 06 22:59:24.287319 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:24.287271 2576 patch_prober.go:28] interesting pod/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 06 22:59:24.287602 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:24.287373 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 06 22:59:25.290376 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:25.290343 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:25.290855 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:25.290714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:25.291413 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:25.291394 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 22:59:30.222190 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:30.222145 2576 patch_prober.go:28] interesting pod/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 22:59:30.224144 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:30.222226 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 22:59:31.258839 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.258807 2576 generic.go:356] "Generic (PLEG): container finished" podID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerID="72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4" exitCode=0 Jul 06 22:59:31.259266 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.258882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerDied","Data":"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4"} Jul 06 22:59:31.259335 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.259317 2576 scope.go:117] "RemoveContainer" containerID="72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4" Jul 06 22:59:31.260368 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.260333 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:31.260470 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.260421 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:31.260557 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:31.260484 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 06 22:59:32.267640 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:32.267599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerStarted","Data":"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478"} Jul 06 22:59:32.456948 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:32.456844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:32.457208 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:32.457177 2576 patch_prober.go:28] interesting pod/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 06 22:59:32.457308 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:32.457243 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 06 22:59:33.457093 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:33.457048 2576 patch_prober.go:28] interesting pod/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 06 22:59:33.457641 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:33.457117 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 06 22:59:34.457534 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:34.457486 2576 patch_prober.go:28] interesting pod/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 06 22:59:34.458158 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:34.458122 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 06 22:59:35.457023 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:35.456947 2576 patch_prober.go:28] interesting pod/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 06 22:59:35.457723 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:35.457275 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 06 22:59:36.460041 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:36.460012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:36.460454 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:36.460278 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:36.461044 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:36.461024 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 22:59:44.182106 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.182059 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 22:59:44.185735 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.185710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.188178 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.188151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"kube-root-ca.crt\"" Jul 06 22:59:44.189138 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.189115 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"openshift-service-ca.crt\"" Jul 06 22:59:44.189138 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.189134 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"default-dockercfg-j5m9j\"" Jul 06 22:59:44.189289 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.189273 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"auth-test-model-c8326f-kserve-self-signed-certs\"" Jul 06 22:59:44.192907 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.192878 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 22:59:44.256387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.256564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.256564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.256564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.256564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.256564 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.256520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzpn\" (UniqueName: \"kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.357879 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.357844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358081 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.357895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358081 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358081 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358230 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzpn\" (UniqueName: \"kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358334 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358554 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358659 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358435 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.358659 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.358590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.360313 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.360292 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.360438 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.360421 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.366253 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.366228 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzpn\" (UniqueName: \"kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn\") pod \"auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.497001 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.496877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:44.635547 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:44.635520 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 22:59:44.637912 ip-10-0-139-147 kubenswrapper[2576]: W0706 22:59:44.637883 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf157d9ac_eb2b_4df8_85d7_f0374de29cc2.slice/crio-5e7beb30cfe7757296ee31672c08758e037f5fc985809b9dd5dbf859648e11a1 WatchSource:0}: Error finding container 5e7beb30cfe7757296ee31672c08758e037f5fc985809b9dd5dbf859648e11a1: Status 404 returned error can't find the container with id 5e7beb30cfe7757296ee31672c08758e037f5fc985809b9dd5dbf859648e11a1 Jul 06 22:59:45.327387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:45.327342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" event={"ID":"f157d9ac-eb2b-4df8-85d7-f0374de29cc2","Type":"ContainerStarted","Data":"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949"} Jul 06 22:59:45.327387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:45.327385 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" event={"ID":"f157d9ac-eb2b-4df8-85d7-f0374de29cc2","Type":"ContainerStarted","Data":"5e7beb30cfe7757296ee31672c08758e037f5fc985809b9dd5dbf859648e11a1"} Jul 06 22:59:45.343838 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:45.343780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" podStartSLOduration=1.343764243 podStartE2EDuration="1.343764243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:59:45.341873284 +0000 UTC m=+2141.368287653" watchObservedRunningTime="2026-07-06 22:59:45.343764243 +0000 UTC m=+2141.370178607" Jul 06 22:59:54.497869 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:54.497826 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:54.498387 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:54.497887 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:54.510263 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:54.510237 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 22:59:55.388489 ip-10-0-139-147 kubenswrapper[2576]: I0706 22:59:55.388410 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 23:00:03.759773 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.759736 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:00:03.763133 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.763116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.766491 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.766463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"default-dockercfg-pcr2j\"" Jul 06 23:00:03.766491 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.766476 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"kube-root-ca.crt\"" Jul 06 23:00:03.766680 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.766519 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"auth-test-model-b0de62-kserve-self-signed-certs\"" Jul 06 23:00:03.766680 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.766523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"openshift-service-ca.crt\"" Jul 06 23:00:03.773506 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.773486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:00:03.834574 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.834763 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.834763 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.834763 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.834895 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.834895 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.834796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26nz\" (UniqueName: \"kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935308 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935308 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935568 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935336 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26nz\" (UniqueName: \"kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935568 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935568 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935568 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935790 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.935851 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.935812 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.936077 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.936057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.937917 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.937893 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.938842 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.938804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:03.944063 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:03.944037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26nz\" (UniqueName: \"kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz\") pod \"auth-test-model-b0de62-kserve-985d97bf9-kxxb9\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:04.073962 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:04.073875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:04.203759 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:04.203734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:00:04.206136 ip-10-0-139-147 kubenswrapper[2576]: W0706 23:00:04.206109 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c575c44_a270_4362_91b9_854fb0a69190.slice/crio-e6feb237eaa1f6d46e664a38a40f2b75b1e8b5ef4245cca4f24bde5fb6ee1049 WatchSource:0}: Error finding container e6feb237eaa1f6d46e664a38a40f2b75b1e8b5ef4245cca4f24bde5fb6ee1049: Status 404 returned error can't find the container with id e6feb237eaa1f6d46e664a38a40f2b75b1e8b5ef4245cca4f24bde5fb6ee1049 Jul 06 23:00:04.418628 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:04.418591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" event={"ID":"8c575c44-a270-4362-91b9-854fb0a69190","Type":"ContainerStarted","Data":"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835"} Jul 06 23:00:04.418628 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:04.418633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" event={"ID":"8c575c44-a270-4362-91b9-854fb0a69190","Type":"ContainerStarted","Data":"e6feb237eaa1f6d46e664a38a40f2b75b1e8b5ef4245cca4f24bde5fb6ee1049"} Jul 06 23:00:04.437084 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:04.437029 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" podStartSLOduration=1.43701314 podStartE2EDuration="1.43701314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:04.435097556 +0000 UTC m=+2160.461511913" watchObservedRunningTime="2026-07-06 23:00:04.43701314 +0000 UTC m=+2160.463427504" Jul 06 23:00:14.074042 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:14.074005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:14.074408 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:14.074059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:14.086110 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:14.086085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:00:14.471846 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:00:14.471819 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:02:00.080515 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:00.080478 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 23:02:01.063192 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:01.063154 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:02:15.101389 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.101340 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" podUID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" containerName="main" containerID="cri-o://06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949" gracePeriod=45 Jul 06 23:02:15.355837 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.355760 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 23:02:15.411464 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411428 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411479 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411597 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411622 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgzpn\" (UniqueName: \"kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn\") pod \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\" (UID: \"f157d9ac-eb2b-4df8-85d7-f0374de29cc2\") " Jul 06 23:02:15.411949 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411797 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:15.411949 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411808 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:15.411949 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.411849 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home" (OuterVolumeSpecName: "home") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:15.413866 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.413834 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:02:15.413866 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.413858 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn" (OuterVolumeSpecName: "kube-api-access-dgzpn") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "kube-api-access-dgzpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:02:15.414063 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.413877 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm" (OuterVolumeSpecName: "dshm") pod "f157d9ac-eb2b-4df8-85d7-f0374de29cc2" (UID: "f157d9ac-eb2b-4df8-85d7-f0374de29cc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:15.512463 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512418 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:15.512463 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512454 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:15.512463 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512465 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:15.512463 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512475 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzpn\" (UniqueName: \"kubernetes.io/projected/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-kube-api-access-dgzpn\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:15.512741 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512485 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:15.512741 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:15.512493 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f157d9ac-eb2b-4df8-85d7-f0374de29cc2-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.001442 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.001408 2576 generic.go:356] "Generic (PLEG): container finished" podID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" containerID="06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949" exitCode=0 Jul 06 23:02:16.001638 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.001453 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" Jul 06 23:02:16.001638 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.001476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" event={"ID":"f157d9ac-eb2b-4df8-85d7-f0374de29cc2","Type":"ContainerDied","Data":"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949"} Jul 06 23:02:16.001638 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.001510 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g" event={"ID":"f157d9ac-eb2b-4df8-85d7-f0374de29cc2","Type":"ContainerDied","Data":"5e7beb30cfe7757296ee31672c08758e037f5fc985809b9dd5dbf859648e11a1"} Jul 06 23:02:16.001638 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.001536 2576 scope.go:117] "RemoveContainer" containerID="06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949" Jul 06 23:02:16.010477 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.010456 2576 scope.go:117] "RemoveContainer" containerID="06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949" Jul 06 23:02:16.010813 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:02:16.010787 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949\": container with ID starting with 06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949 not found: ID does not exist" containerID="06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949" Jul 06 23:02:16.010923 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.010824 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949"} err="failed to get container status \"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949\": rpc error: code = NotFound desc = could not find container \"06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949\": container with ID starting with 06249bd7cc3a837bf5cbe6278ff1a5a2c0c8314be38d4889080d591985d5b949 not found: ID does not exist" Jul 06 23:02:16.022230 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.022194 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 23:02:16.026042 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.026010 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/auth-test-model-c8326f-kserve-84db5b4d8c-wqb2g"] Jul 06 23:02:16.084343 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.084277 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" podUID="8c575c44-a270-4362-91b9-854fb0a69190" containerName="main" containerID="cri-o://4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835" gracePeriod=45 Jul 06 23:02:16.346534 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.346504 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:02:16.421202 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421406 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421211 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r26nz\" (UniqueName: \"kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421406 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421406 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421316 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421406 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421383 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421647 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421410 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home\") pod \"8c575c44-a270-4362-91b9-854fb0a69190\" (UID: \"8c575c44-a270-4362-91b9-854fb0a69190\") " Jul 06 23:02:16.421647 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421502 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:16.421756 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421668 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:16.421756 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421699 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.421756 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.421712 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home" (OuterVolumeSpecName: "home") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:16.423732 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.423700 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:02:16.423732 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.423698 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm" (OuterVolumeSpecName: "dshm") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:16.423896 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.423740 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz" (OuterVolumeSpecName: "kube-api-access-r26nz") pod "8c575c44-a270-4362-91b9-854fb0a69190" (UID: "8c575c44-a270-4362-91b9-854fb0a69190"). InnerVolumeSpecName "kube-api-access-r26nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:02:16.522811 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.522723 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26nz\" (UniqueName: \"kubernetes.io/projected/8c575c44-a270-4362-91b9-854fb0a69190-kube-api-access-r26nz\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.522811 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.522801 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c575c44-a270-4362-91b9-854fb0a69190-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.522811 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.522812 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.523048 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.522819 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.523048 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.522827 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c575c44-a270-4362-91b9-854fb0a69190-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:02:16.523048 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:16.523008 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" path="/var/lib/kubelet/pods/f157d9ac-eb2b-4df8-85d7-f0374de29cc2/volumes" Jul 06 23:02:17.007434 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.007397 2576 generic.go:356] "Generic (PLEG): container finished" podID="8c575c44-a270-4362-91b9-854fb0a69190" containerID="4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835" exitCode=0 Jul 06 23:02:17.007657 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.007457 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" Jul 06 23:02:17.007657 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.007480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" event={"ID":"8c575c44-a270-4362-91b9-854fb0a69190","Type":"ContainerDied","Data":"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835"} Jul 06 23:02:17.007657 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.007533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9" event={"ID":"8c575c44-a270-4362-91b9-854fb0a69190","Type":"ContainerDied","Data":"e6feb237eaa1f6d46e664a38a40f2b75b1e8b5ef4245cca4f24bde5fb6ee1049"} Jul 06 23:02:17.007657 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.007560 2576 scope.go:117] "RemoveContainer" containerID="4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835" Jul 06 23:02:17.016264 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.016239 2576 scope.go:117] "RemoveContainer" containerID="4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835" Jul 06 23:02:17.016616 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:02:17.016595 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835\": container with ID starting with 4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835 not found: ID does not exist" containerID="4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835" Jul 06 23:02:17.016687 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.016628 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835"} err="failed to get container status \"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835\": rpc error: code = NotFound desc = could not find container \"4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835\": container with ID starting with 4e48ade14387cad731aa53f03625e26ea4e031c758796da853f6850bab3e4835 not found: ID does not exist" Jul 06 23:02:17.025893 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.025856 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:02:17.029350 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:17.029317 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/auth-test-model-b0de62-kserve-985d97bf9-kxxb9"] Jul 06 23:02:18.523059 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:18.523017 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c575c44-a270-4362-91b9-854fb0a69190" path="/var/lib/kubelet/pods/8c575c44-a270-4362-91b9-854fb0a69190/volumes" Jul 06 23:02:29.872905 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.872863 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873265 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" containerName="main" Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873283 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" containerName="main" Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873303 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c575c44-a270-4362-91b9-854fb0a69190" containerName="main" Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873308 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c575c44-a270-4362-91b9-854fb0a69190" containerName="main" Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873370 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f157d9ac-eb2b-4df8-85d7-f0374de29cc2" containerName="main" Jul 06 23:02:29.873379 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.873381 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c575c44-a270-4362-91b9-854fb0a69190" containerName="main" Jul 06 23:02:29.878264 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.878241 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:29.881724 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.881670 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"openshift-service-ca.crt\"" Jul 06 23:02:29.881724 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.881670 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"kube-root-ca.crt\"" Jul 06 23:02:29.881724 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.881711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"sub-test-model-b0de62-kserve-self-signed-certs\"" Jul 06 23:02:29.881969 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.881670 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b0de62\"/\"default-dockercfg-pcr2j\"" Jul 06 23:02:29.885327 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:29.885290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:02:30.040027 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.039957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrt6\" (UniqueName: \"kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.040204 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.040032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.040204 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.040093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.040204 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.040165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.040204 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.040198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.040347 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.040248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141613 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrt6\" (UniqueName: \"kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141613 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141613 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141844 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141844 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.141844 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.141671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.142095 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.142072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.142095 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.142086 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.142185 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.142151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.144107 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.144090 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.144182 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.144165 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.148931 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.148911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrt6\" (UniqueName: \"kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6\") pod \"sub-test-model-b0de62-kserve-68d57978d5-8gj78\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.189141 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.189102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:30.331244 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:30.331191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:02:30.334115 ip-10-0-139-147 kubenswrapper[2576]: W0706 23:02:30.334082 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb34016_db4a_4d81_9693_60f1200570b0.slice/crio-fb497874b81c1969f5498535aeddbdfb35eec2e1d4f3f35bb98dd3faec43a0f6 WatchSource:0}: Error finding container fb497874b81c1969f5498535aeddbdfb35eec2e1d4f3f35bb98dd3faec43a0f6: Status 404 returned error can't find the container with id fb497874b81c1969f5498535aeddbdfb35eec2e1d4f3f35bb98dd3faec43a0f6 Jul 06 23:02:31.077789 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:31.077744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" event={"ID":"cfb34016-db4a-4d81-9693-60f1200570b0","Type":"ContainerStarted","Data":"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66"} Jul 06 23:02:31.078226 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:31.077796 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" event={"ID":"cfb34016-db4a-4d81-9693-60f1200570b0","Type":"ContainerStarted","Data":"fb497874b81c1969f5498535aeddbdfb35eec2e1d4f3f35bb98dd3faec43a0f6"} Jul 06 23:02:31.094312 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:31.094257 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" podStartSLOduration=2.094238792 podStartE2EDuration="2.094238792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:02:31.093228773 +0000 UTC m=+2307.119643142" watchObservedRunningTime="2026-07-06 23:02:31.094238792 +0000 UTC m=+2307.120653155" Jul 06 23:02:40.190192 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:40.190147 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:40.190610 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:40.190206 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:40.203093 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:40.203057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:02:41.133946 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:02:41.133917 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:03:12.552539 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:12.552503 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:03:17.982849 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.982819 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:03:17.986127 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.986111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:17.988743 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.988726 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"openshift-service-ca.crt\"" Jul 06 23:03:17.989848 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.989823 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"kube-root-ca.crt\"" Jul 06 23:03:17.989987 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.989882 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"rate-test-model-c8326f-kserve-self-signed-certs\"" Jul 06 23:03:17.989987 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.989902 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8326f\"/\"default-dockercfg-j5m9j\"" Jul 06 23:03:17.996951 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:17.996927 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:03:18.059415 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.059415 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9cbf\" (UniqueName: \"kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.059632 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.059632 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.059632 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.059757 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.059647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.160996 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.160934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161190 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161190 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161190 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9cbf\" (UniqueName: \"kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161190 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161354 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.161484 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.162079 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.161962 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.162292 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.162267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.164337 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.164314 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.164717 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.164696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.171278 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.171255 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9cbf\" (UniqueName: \"kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf\") pod \"rate-test-model-c8326f-kserve-644d5ffd59-4qkgt\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.295120 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.295022 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:18.430278 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:18.430245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:03:18.433252 ip-10-0-139-147 kubenswrapper[2576]: W0706 23:03:18.433220 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9be9478a_bf7e_4527_9a03_169f8ec17618.slice/crio-9f7588f35a15ae7f8d8629d82162d6319d7f2e5e338c627223eaf5c34707263f WatchSource:0}: Error finding container 9f7588f35a15ae7f8d8629d82162d6319d7f2e5e338c627223eaf5c34707263f: Status 404 returned error can't find the container with id 9f7588f35a15ae7f8d8629d82162d6319d7f2e5e338c627223eaf5c34707263f Jul 06 23:03:19.304382 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:19.304348 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" event={"ID":"9be9478a-bf7e-4527-9a03-169f8ec17618","Type":"ContainerStarted","Data":"7198f65961f9d6731c610c9068bb93bd72a1a5f8485ded432e6cbb29d2e04dc3"} Jul 06 23:03:19.304382 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:19.304386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" event={"ID":"9be9478a-bf7e-4527-9a03-169f8ec17618","Type":"ContainerStarted","Data":"9f7588f35a15ae7f8d8629d82162d6319d7f2e5e338c627223eaf5c34707263f"} Jul 06 23:03:19.321793 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:19.321743 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" podStartSLOduration=2.321728667 podStartE2EDuration="2.321728667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:03:19.320494777 +0000 UTC m=+2355.346909158" watchObservedRunningTime="2026-07-06 23:03:19.321728667 +0000 UTC m=+2355.348143030" Jul 06 23:03:27.572866 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.572825 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" podUID="cfb34016-db4a-4d81-9693-60f1200570b0" containerName="main" containerID="cri-o://ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66" gracePeriod=45 Jul 06 23:03:27.826734 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.826671 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:03:27.950790 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.950737 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.950790 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.950793 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.951071 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.950817 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rrt6\" (UniqueName: \"kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.951071 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.950842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.951071 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.950868 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.951071 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951018 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs\") pod \"cfb34016-db4a-4d81-9693-60f1200570b0\" (UID: \"cfb34016-db4a-4d81-9693-60f1200570b0\") " Jul 06 23:03:27.951260 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951115 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:27.951260 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951172 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home" (OuterVolumeSpecName: "home") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:27.951260 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951187 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:27.951423 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951367 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:27.951423 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951381 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:27.951423 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.951389 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:27.953147 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.953112 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm" (OuterVolumeSpecName: "dshm") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:27.953147 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.953124 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6" (OuterVolumeSpecName: "kube-api-access-9rrt6") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "kube-api-access-9rrt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:03:27.953299 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:27.953152 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfb34016-db4a-4d81-9693-60f1200570b0" (UID: "cfb34016-db4a-4d81-9693-60f1200570b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:03:28.052385 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.052334 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb34016-db4a-4d81-9693-60f1200570b0-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:28.052385 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.052378 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb34016-db4a-4d81-9693-60f1200570b0-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:28.052385 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.052389 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrt6\" (UniqueName: \"kubernetes.io/projected/cfb34016-db4a-4d81-9693-60f1200570b0-kube-api-access-9rrt6\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:03:28.296744 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.296695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:28.296744 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.296753 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:28.309521 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.309493 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:28.343877 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.343845 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfb34016-db4a-4d81-9693-60f1200570b0" containerID="ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66" exitCode=0 Jul 06 23:03:28.344055 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.343884 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" Jul 06 23:03:28.344055 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.343931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" event={"ID":"cfb34016-db4a-4d81-9693-60f1200570b0","Type":"ContainerDied","Data":"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66"} Jul 06 23:03:28.344055 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.343971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78" event={"ID":"cfb34016-db4a-4d81-9693-60f1200570b0","Type":"ContainerDied","Data":"fb497874b81c1969f5498535aeddbdfb35eec2e1d4f3f35bb98dd3faec43a0f6"} Jul 06 23:03:28.344055 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.344002 2576 scope.go:117] "RemoveContainer" containerID="ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66" Jul 06 23:03:28.352546 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.352524 2576 scope.go:117] "RemoveContainer" containerID="ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66" Jul 06 23:03:28.352819 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:03:28.352795 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66\": container with ID starting with ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66 not found: ID does not exist" containerID="ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66" Jul 06 23:03:28.352864 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.352831 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66"} err="failed to get container status \"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66\": rpc error: code = NotFound desc = could not find container \"ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66\": container with ID starting with ab18667d7b03860137634ed1443d38ab6462a9e9459b0a9c0df58efcc7f87e66 not found: ID does not exist" Jul 06 23:03:28.356208 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.356187 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:03:28.363716 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.363697 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:03:28.367433 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.367414 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b0de62/sub-test-model-b0de62-kserve-68d57978d5-8gj78"] Jul 06 23:03:28.528570 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:03:28.528536 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb34016-db4a-4d81-9693-60f1200570b0" path="/var/lib/kubelet/pods/cfb34016-db4a-4d81-9693-60f1200570b0/volumes" Jul 06 23:04:19.380348 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:19.380313 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:04:34.400072 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.400034 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" podUID="9be9478a-bf7e-4527-9a03-169f8ec17618" containerName="main" containerID="cri-o://7198f65961f9d6731c610c9068bb93bd72a1a5f8485ded432e6cbb29d2e04dc3" gracePeriod=45 Jul 06 23:04:34.621692 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.621656 2576 generic.go:356] "Generic (PLEG): container finished" podID="9be9478a-bf7e-4527-9a03-169f8ec17618" containerID="7198f65961f9d6731c610c9068bb93bd72a1a5f8485ded432e6cbb29d2e04dc3" exitCode=0 Jul 06 23:04:34.621860 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.621738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" event={"ID":"9be9478a-bf7e-4527-9a03-169f8ec17618","Type":"ContainerDied","Data":"7198f65961f9d6731c610c9068bb93bd72a1a5f8485ded432e6cbb29d2e04dc3"} Jul 06 23:04:34.662109 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.662043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:04:34.741926 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.741885 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742159 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.741959 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9cbf\" (UniqueName: \"kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742159 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742046 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742159 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742076 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742159 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742159 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache\") pod \"9be9478a-bf7e-4527-9a03-169f8ec17618\" (UID: \"9be9478a-bf7e-4527-9a03-169f8ec17618\") " Jul 06 23:04:34.742419 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742307 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home" (OuterVolumeSpecName: "home") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:04:34.742419 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742357 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:04:34.742419 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742416 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-home\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:34.742577 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742432 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-tmp-dir\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:34.742577 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.742501 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache" (OuterVolumeSpecName: "model-cache") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:04:34.744341 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.744312 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:04:34.744662 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.744636 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm" (OuterVolumeSpecName: "dshm") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:04:34.744662 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.744652 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf" (OuterVolumeSpecName: "kube-api-access-p9cbf") pod "9be9478a-bf7e-4527-9a03-169f8ec17618" (UID: "9be9478a-bf7e-4527-9a03-169f8ec17618"). InnerVolumeSpecName "kube-api-access-p9cbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:04:34.843263 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.843226 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9be9478a-bf7e-4527-9a03-169f8ec17618-tls-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:34.843263 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.843256 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9cbf\" (UniqueName: \"kubernetes.io/projected/9be9478a-bf7e-4527-9a03-169f8ec17618-kube-api-access-p9cbf\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:34.843263 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.843266 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-dshm\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:34.843503 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:34.843274 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9be9478a-bf7e-4527-9a03-169f8ec17618-model-cache\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:04:35.627434 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:35.627399 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" event={"ID":"9be9478a-bf7e-4527-9a03-169f8ec17618","Type":"ContainerDied","Data":"9f7588f35a15ae7f8d8629d82162d6319d7f2e5e338c627223eaf5c34707263f"} Jul 06 23:04:35.627434 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:35.627430 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt" Jul 06 23:04:35.628014 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:35.627450 2576 scope.go:117] "RemoveContainer" containerID="7198f65961f9d6731c610c9068bb93bd72a1a5f8485ded432e6cbb29d2e04dc3" Jul 06 23:04:35.648093 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:35.648068 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:04:35.653512 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:35.653490 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c8326f/rate-test-model-c8326f-kserve-644d5ffd59-4qkgt"] Jul 06 23:04:36.523370 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:04:36.523331 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be9478a-bf7e-4527-9a03-169f8ec17618" path="/var/lib/kubelet/pods/9be9478a-bf7e-4527-9a03-169f8ec17618/volumes" Jul 06 23:05:04.277295 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:04.277256 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 23:05:04.277823 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:04.277534 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" containerID="cri-o://a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35" gracePeriod=30 Jul 06 23:05:09.532373 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.532344 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 23:05:09.649903 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.649870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650126 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.649922 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650126 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.649954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650126 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvhfd\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650290 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:09.650321 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650313 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token\") pod \"3b1fe7c6-994a-4764-ac30-82e625c09e66\" (UID: \"3b1fe7c6-994a-4764-ac30-82e625c09e66\") " Jul 06 23:05:09.650612 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650325 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:09.650612 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650451 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:05:09.650612 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650535 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:09.650779 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650754 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.650838 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650781 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b1fe7c6-994a-4764-ac30-82e625c09e66-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.650838 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650795 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.650838 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.650812 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.651310 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.651288 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:09.652751 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.652724 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:05:09.652851 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.652768 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:09.652851 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.652782 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd" (OuterVolumeSpecName: "kube-api-access-rvhfd") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "kube-api-access-rvhfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:05:09.652851 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.652797 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b1fe7c6-994a-4764-ac30-82e625c09e66" (UID: "3b1fe7c6-994a-4764-ac30-82e625c09e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:05:09.751350 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.751319 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvhfd\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-kube-api-access-rvhfd\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.751350 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.751347 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.751350 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.751356 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.751585 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.751364 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.751585 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.751372 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b1fe7c6-994a-4764-ac30-82e625c09e66-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:09.777457 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.777420 2576 generic.go:356] "Generic (PLEG): container finished" podID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerID="a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35" exitCode=0 Jul 06 23:05:09.777601 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.777475 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" Jul 06 23:05:09.777601 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.777504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerDied","Data":"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35"} Jul 06 23:05:09.777601 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.777559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b" event={"ID":"3b1fe7c6-994a-4764-ac30-82e625c09e66","Type":"ContainerDied","Data":"d7c63bf65a14e10697ae02ea4d377dbecff59cfd96588f4345bca6290f06ac5f"} Jul 06 23:05:09.777601 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.777588 2576 scope.go:117] "RemoveContainer" containerID="a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35" Jul 06 23:05:09.786397 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.786377 2576 scope.go:117] "RemoveContainer" containerID="4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b" Jul 06 23:05:09.795129 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.795109 2576 scope.go:117] "RemoveContainer" containerID="a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35" Jul 06 23:05:09.795459 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:05:09.795369 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35\": container with ID starting with a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35 not found: ID does not exist" containerID="a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35" Jul 06 23:05:09.795459 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.795399 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35"} err="failed to get container status \"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35\": rpc error: code = NotFound desc = could not find container \"a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35\": container with ID starting with a24a4b57efd1d0b09a7916d9c7ff0615abe8e5d7dcde195703e8f61be220ac35 not found: ID does not exist" Jul 06 23:05:09.795459 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.795423 2576 scope.go:117] "RemoveContainer" containerID="4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b" Jul 06 23:05:09.795691 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:05:09.795671 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b\": container with ID starting with 4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b not found: ID does not exist" containerID="4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b" Jul 06 23:05:09.795747 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.795700 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b"} err="failed to get container status \"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b\": rpc error: code = NotFound desc = could not find container \"4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b\": container with ID starting with 4ba2ce95526c8841f74e7c65b0327356f3212bf41390a188b1da7b551ffe659b not found: ID does not exist" Jul 06 23:05:09.797414 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.797394 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 23:05:09.801272 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:09.801249 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c8326f-openshift-default-7b56fcbf8c-bqd7b"] Jul 06 23:05:10.520855 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:10.520823 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" path="/var/lib/kubelet/pods/3b1fe7c6-994a-4764-ac30-82e625c09e66/volumes" Jul 06 23:05:11.784095 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:11.784063 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 23:05:11.784576 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:11.784315 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" containerID="cri-o://2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478" gracePeriod=30 Jul 06 23:05:12.507684 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:12.507651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ff4d58f8-cv6w6_7b765adc-3a35-4595-a134-52eeb7b5d194/authorino/0.log" Jul 06 23:05:17.033484 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.033460 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 23:05:17.112151 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112063 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112151 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112101 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112151 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112261 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x792\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112391 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112427 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112423 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket\") pod \"d931979e-beaf-45a2-adad-c57afd9ebfee\" (UID: \"d931979e-beaf-45a2-adad-c57afd9ebfee\") " Jul 06 23:05:17.112729 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112436 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:17.112729 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112613 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:17.112729 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112675 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-credential-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.112729 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112689 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 23:05:17.112939 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.112873 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:17.113100 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.113077 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data" (OuterVolumeSpecName: "istio-data") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:17.114549 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.114515 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token" (OuterVolumeSpecName: "istio-token") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:05:17.114771 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.114728 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792" (OuterVolumeSpecName: "kube-api-access-9x792") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "kube-api-access-9x792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:05:17.114854 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.114789 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:05:17.114854 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.114814 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d931979e-beaf-45a2-adad-c57afd9ebfee" (UID: "d931979e-beaf-45a2-adad-c57afd9ebfee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 23:05:17.213365 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213328 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-socket\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213365 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213357 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-envoy\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213365 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213367 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-token\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213365 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213375 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-data\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213650 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213384 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x792\" (UniqueName: \"kubernetes.io/projected/d931979e-beaf-45a2-adad-c57afd9ebfee-kube-api-access-9x792\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213650 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213392 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d931979e-beaf-45a2-adad-c57afd9ebfee-istiod-ca-cert\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213650 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213400 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d931979e-beaf-45a2-adad-c57afd9ebfee-workload-certs\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.213650 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.213408 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d931979e-beaf-45a2-adad-c57afd9ebfee-istio-podinfo\") on node \"ip-10-0-139-147.ec2.internal\" DevicePath \"\"" Jul 06 23:05:17.818826 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.818790 2576 generic.go:356] "Generic (PLEG): container finished" podID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerID="2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478" exitCode=0 Jul 06 23:05:17.819014 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.818860 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" Jul 06 23:05:17.819014 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.818870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerDied","Data":"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478"} Jul 06 23:05:17.819014 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.818961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6" event={"ID":"d931979e-beaf-45a2-adad-c57afd9ebfee","Type":"ContainerDied","Data":"bbc19156be3dbc74c539c90a2330f7af423f2190eff15f07182b484ac4b995a1"} Jul 06 23:05:17.819014 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.819014 2576 scope.go:117] "RemoveContainer" containerID="2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478" Jul 06 23:05:17.827952 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.827930 2576 scope.go:117] "RemoveContainer" containerID="72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4" Jul 06 23:05:17.835673 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.835655 2576 scope.go:117] "RemoveContainer" containerID="2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478" Jul 06 23:05:17.835928 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:05:17.835907 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478\": container with ID starting with 2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478 not found: ID does not exist" containerID="2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478" Jul 06 23:05:17.836013 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.835942 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478"} err="failed to get container status \"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478\": rpc error: code = NotFound desc = could not find container \"2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478\": container with ID starting with 2f8bb9241a0921ee90162c1c96df6e27423f380a0bd4752ec1b2bfdec39f5478 not found: ID does not exist" Jul 06 23:05:17.836013 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.835967 2576 scope.go:117] "RemoveContainer" containerID="72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4" Jul 06 23:05:17.836241 ip-10-0-139-147 kubenswrapper[2576]: E0706 23:05:17.836225 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4\": container with ID starting with 72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4 not found: ID does not exist" containerID="72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4" Jul 06 23:05:17.836292 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.836246 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4"} err="failed to get container status \"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4\": rpc error: code = NotFound desc = could not find container \"72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4\": container with ID starting with 72a175d4d45041b96ce3eed2786c30ae5d712069f2629aeea5f52827ea81cfb4 not found: ID does not exist" Jul 06 23:05:17.839587 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.839561 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 23:05:17.842834 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.842811 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b0de62-openshift-default-675d7484d7-55sv6"] Jul 06 23:05:17.944079 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:17.944049 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jl2v8_0921b309-f093-4b4e-a8a7-0cdf615c0476/manager/0.log" Jul 06 23:05:18.297100 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:18.297073 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcf54485-vrls4_7ed5ae59-844f-4123-904a-86f9802ef890/manager/0.log" Jul 06 23:05:18.303539 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:18.303517 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcf54485-vrls4_7ed5ae59-844f-4123-904a-86f9802ef890/copy-manifests/0.log" Jul 06 23:05:18.521095 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:18.521057 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" path="/var/lib/kubelet/pods/d931979e-beaf-45a2-adad-c57afd9ebfee/volumes" Jul 06 23:05:19.684265 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:19.684237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ff4d58f8-cv6w6_7b765adc-3a35-4595-a134-52eeb7b5d194/authorino/0.log" Jul 06 23:05:20.874429 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:20.874403 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678tffjd_1d747c0e-f59e-4b18-84d6-26dafab9d7e5/istio-proxy/0.log" Jul 06 23:05:21.200615 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:21.200538 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6975f4cc-vljvp_b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3/kube-auth-proxy/0.log" Jul 06 23:05:21.434575 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:21.434548 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7mbw6_9b0db4a9-e5be-4c04-9457-0ef3ff4e7275/payload-pre-processing/0.log" Jul 06 23:05:22.138857 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:22.138828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-q8b2f_1e972134-ce83-4459-a5ba-d85be2b6a40d/main/0.log" Jul 06 23:05:22.252830 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:22.252801 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xkrf9_c5ef5f2e-8037-420f-b911-1abbd7ce02b2/main/0.log" Jul 06 23:05:22.603577 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:22.603498 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-gkm88_07c24cee-1577-4e97-acd7-6cb94c3340e4/main/0.log" Jul 06 23:05:29.694033 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:29.694001 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-plfpg_9b3f4825-2adc-4709-b51c-1cdb45711326/global-pull-secret-syncer/0.log" Jul 06 23:05:29.784458 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:29.784426 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t6bsh_a1cb0358-d19d-4f0a-9bf3-22d6d864a149/konnectivity-agent/0.log" Jul 06 23:05:29.880215 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:29.880188 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-147.ec2.internal_ca8eaf696cc1052bc992b005a70346d1/haproxy/0.log" Jul 06 23:05:34.990263 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:34.990227 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77ff4d58f8-cv6w6_7b765adc-3a35-4595-a134-52eeb7b5d194/authorino/0.log" Jul 06 23:05:37.500564 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:37.500535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5q64_7af163fd-380c-40c1-8bf5-09ebf2b7ac36/node-exporter/0.log" Jul 06 23:05:37.521858 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:37.521833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5q64_7af163fd-380c-40c1-8bf5-09ebf2b7ac36/kube-rbac-proxy/0.log" Jul 06 23:05:37.544256 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:37.544229 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5q64_7af163fd-380c-40c1-8bf5-09ebf2b7ac36/init-textfile/0.log" Jul 06 23:05:39.956867 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.956823 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps"] Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957174 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957187 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957198 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957204 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957212 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be9478a-bf7e-4527-9a03-169f8ec17618" containerName="main" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957217 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be9478a-bf7e-4527-9a03-169f8ec17618" containerName="main" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957242 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb34016-db4a-4d81-9693-60f1200570b0" containerName="main" Jul 06 23:05:39.957298 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957247 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb34016-db4a-4d81-9693-60f1200570b0" containerName="main" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957316 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957328 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb34016-db4a-4d81-9693-60f1200570b0" containerName="main" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957338 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1fe7c6-994a-4764-ac30-82e625c09e66" containerName="istio-proxy" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957346 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957355 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be9478a-bf7e-4527-9a03-169f8ec17618" containerName="main" Jul 06 23:05:39.957622 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.957363 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d931979e-beaf-45a2-adad-c57afd9ebfee" containerName="istio-proxy" Jul 06 23:05:39.962221 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.962200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:39.964447 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.964422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8lsnx\"/\"kube-root-ca.crt\"" Jul 06 23:05:39.964575 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.964513 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8lsnx\"/\"openshift-service-ca.crt\"" Jul 06 23:05:39.965433 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.965414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8lsnx\"/\"default-dockercfg-h582k\"" Jul 06 23:05:39.967563 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:39.967537 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps"] Jul 06 23:05:40.003943 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.003910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-proc\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.004209 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.003947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhhs\" (UniqueName: \"kubernetes.io/projected/cb3959b4-3136-43c9-b28e-2d455362d7b4-kube-api-access-xxhhs\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.004209 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.003999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-sys\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.004209 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.004094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-lib-modules\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.004209 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.004155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-podres\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105333 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-podres\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-proc\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhhs\" (UniqueName: \"kubernetes.io/projected/cb3959b4-3136-43c9-b28e-2d455362d7b4-kube-api-access-xxhhs\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-sys\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-lib-modules\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-proc\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105517 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-sys\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105535 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-podres\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.105591 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.105561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3959b4-3136-43c9-b28e-2d455362d7b4-lib-modules\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.113722 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.113694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhhs\" (UniqueName: \"kubernetes.io/projected/cb3959b4-3136-43c9-b28e-2d455362d7b4-kube-api-access-xxhhs\") pod \"perf-node-gather-daemonset-w28ps\" (UID: \"cb3959b4-3136-43c9-b28e-2d455362d7b4\") " pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.273150 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.273055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.400638 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.400594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps"] Jul 06 23:05:40.406516 ip-10-0-139-147 kubenswrapper[2576]: W0706 23:05:40.406479 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb3959b4_3136_43c9_b28e_2d455362d7b4.slice/crio-2e48a9714d8d008c4c90a7847a653e61d4fe83e5552ef1f8232b5bf4bd7af9e5 WatchSource:0}: Error finding container 2e48a9714d8d008c4c90a7847a653e61d4fe83e5552ef1f8232b5bf4bd7af9e5: Status 404 returned error can't find the container with id 2e48a9714d8d008c4c90a7847a653e61d4fe83e5552ef1f8232b5bf4bd7af9e5 Jul 06 23:05:40.915509 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.915475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" event={"ID":"cb3959b4-3136-43c9-b28e-2d455362d7b4","Type":"ContainerStarted","Data":"e571c08a3651cb13da707983215af3cb557d4782757881e5bc4fa2180a29d771"} Jul 06 23:05:40.915701 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.915528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" event={"ID":"cb3959b4-3136-43c9-b28e-2d455362d7b4","Type":"ContainerStarted","Data":"2e48a9714d8d008c4c90a7847a653e61d4fe83e5552ef1f8232b5bf4bd7af9e5"} Jul 06 23:05:40.915701 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.915594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:40.930443 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:40.930391 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" podStartSLOduration=1.93037628 podStartE2EDuration="1.93037628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:05:40.92955792 +0000 UTC m=+2496.955972288" watchObservedRunningTime="2026-07-06 23:05:40.93037628 +0000 UTC m=+2496.956790643" Jul 06 23:05:42.266071 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:42.266044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlk95_e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e/dns/0.log" Jul 06 23:05:42.286118 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:42.286083 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlk95_e72aeaae-0cf8-4cbf-bab6-4d390c8d4c6e/kube-rbac-proxy/0.log" Jul 06 23:05:42.383447 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:42.383422 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xq8p6_c8479ad7-5197-44e5-b8a0-f6d111714378/dns-node-resolver/0.log" Jul 06 23:05:42.982688 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:42.982663 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xhdwp_9a39f0e7-e0e5-4107-afcb-86938748c51b/node-ca/0.log" Jul 06 23:05:43.914634 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:43.914604 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678tffjd_1d747c0e-f59e-4b18-84d6-26dafab9d7e5/istio-proxy/0.log" Jul 06 23:05:44.070497 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:44.070465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6975f4cc-vljvp_b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3/kube-auth-proxy/0.log" Jul 06 23:05:44.191716 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:44.191634 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7mbw6_9b0db4a9-e5be-4c04-9457-0ef3ff4e7275/payload-pre-processing/0.log" Jul 06 23:05:44.979127 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:44.979096 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ntp85_e70472f5-e8bd-4ddb-9ea4-0c4559febb51/serve-healthcheck-canary/0.log" Jul 06 23:05:45.467560 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:45.467529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m8tk_8c5692a4-c888-4fd7-82b4-184d1b1c1569/kube-rbac-proxy/0.log" Jul 06 23:05:45.489461 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:45.489436 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m8tk_8c5692a4-c888-4fd7-82b4-184d1b1c1569/exporter/0.log" Jul 06 23:05:45.509735 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:45.509713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7m8tk_8c5692a4-c888-4fd7-82b4-184d1b1c1569/extractor/0.log" Jul 06 23:05:46.929858 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:46.929771 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8lsnx/perf-node-gather-daemonset-w28ps" Jul 06 23:05:48.128448 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:48.128412 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jl2v8_0921b309-f093-4b4e-a8a7-0cdf615c0476/manager/0.log" Jul 06 23:05:48.291851 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:48.291823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcf54485-vrls4_7ed5ae59-844f-4123-904a-86f9802ef890/manager/0.log" Jul 06 23:05:48.311283 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:48.311258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcf54485-vrls4_7ed5ae59-844f-4123-904a-86f9802ef890/copy-manifests/0.log" Jul 06 23:05:49.913288 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:49.913257 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67b7b4db78-jg9lv_cd1aeaea-f6f1-457c-b337-b5b64288f8ee/manager/0.log" Jul 06 23:05:50.240730 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:50.240671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678tffjd_1d747c0e-f59e-4b18-84d6-26dafab9d7e5/istio-proxy/0.log" Jul 06 23:05:50.375041 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:50.375014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6975f4cc-vljvp_b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3/kube-auth-proxy/0.log" Jul 06 23:05:50.479366 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:50.479337 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7mbw6_9b0db4a9-e5be-4c04-9457-0ef3ff4e7275/payload-pre-processing/0.log" Jul 06 23:05:51.675429 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:51.675347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678tffjd_1d747c0e-f59e-4b18-84d6-26dafab9d7e5/istio-proxy/0.log" Jul 06 23:05:51.814956 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:51.814931 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6975f4cc-vljvp_b2ab1f96-8212-4f0e-afeb-6c20f0ef73f3/kube-auth-proxy/0.log" Jul 06 23:05:51.913893 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:51.913848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-7mbw6_9b0db4a9-e5be-4c04-9457-0ef3ff4e7275/payload-pre-processing/0.log" Jul 06 23:05:55.031438 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:55.031399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bjqkp_ad3264a6-1ef1-49d8-a9cf-3d479eb1466b/migrator/0.log" Jul 06 23:05:55.052316 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:55.052287 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bjqkp_ad3264a6-1ef1-49d8-a9cf-3d479eb1466b/graceful-termination/0.log" Jul 06 23:05:56.528527 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.528499 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/kube-multus-additional-cni-plugins/0.log" Jul 06 23:05:56.547797 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.547768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/egress-router-binary-copy/0.log" Jul 06 23:05:56.567693 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.567665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/cni-plugins/0.log" Jul 06 23:05:56.586719 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.586699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/bond-cni-plugin/0.log" Jul 06 23:05:56.605501 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.605480 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/routeoverride-cni/0.log" Jul 06 23:05:56.625311 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.625281 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/whereabouts-cni-bincopy/0.log" Jul 06 23:05:56.644558 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.644533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j29rx_89cdd929-92a4-450e-a67a-ebc2259983ec/whereabouts-cni/0.log" Jul 06 23:05:56.831686 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.831602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l75jp_e8067716-03ba-4d64-bfd8-ed81e0aa6ff4/kube-multus/0.log" Jul 06 23:05:56.964363 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.964322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2h59_f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd/network-metrics-daemon/0.log" Jul 06 23:05:56.983598 ip-10-0-139-147 kubenswrapper[2576]: I0706 23:05:56.983570 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2h59_f5d5b891-7c15-4ecd-a6f9-8b7f0e941cbd/kube-rbac-proxy/0.log"