Jul 08 16:32:06.487895 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 16:32:06.487912 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 16:32:06.487922 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 16:32:06.488276 ip-10-0-130-182 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 16:32:16.452498 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 08 16:32:16.452513 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6a9dad09279242979863ae4c66f76ebb --
Jul 08 16:34:09.679885 ip-10-0-130-182 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:34:10.008953 ip-10-0-130-182 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:10.008953 ip-10-0-130-182 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:34:10.008953 ip-10-0-130-182 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:10.008953 ip-10-0-130-182 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:34:10.008953 ip-10-0-130-182 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:10.009730 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.009614    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:34:10.013922 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013904    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.013922 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013922    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013927    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013931    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013934    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013936    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013939    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013942    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013944    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013947    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013950    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013952    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013955    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013958    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013961    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013963    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013966    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013968    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013971    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013984    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.013992 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013987    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013990    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013992    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013995    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.013998    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014001    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014003    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014006    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014008    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014011    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014013    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014016    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014019    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014021    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014023    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014026    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014028    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014030    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014033    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014035    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.014497 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014038    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014040    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014043    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014046    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014049    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014052    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014054    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014056    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014059    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014061    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014063    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014065    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014068    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014071    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014074    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014077    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014079    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014082    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014084    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014087    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.015013 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014091    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014094    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014098    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014100    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014103    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014105    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014108    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014111    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014113    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014116    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014118    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014120    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014123    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014125    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014127    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014130    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014132    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014135    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014137    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014139    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.015507 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014141    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014144    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014146    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014148    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014152    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014155    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014159    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014162    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014164    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014167    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014169    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014172    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014174    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014177    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014179    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014181    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014184    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014186    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014190    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014924    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.016005 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014931    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014934    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014938    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014941    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014944    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014947    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014949    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014952    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014954    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014957    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014959    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014962    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014964    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014966    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014969    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014971    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014974    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014976    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014987    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.016516 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014991    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014995    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.014999    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015002    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015005    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015009    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015013    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015016    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015019    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015021    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015024    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015027    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015029    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015032    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015034    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015037    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015039    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015041    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015044    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.017041 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015047    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015050    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015052    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015056    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015058    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015062    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015064    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015067    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015070    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015072    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015074    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015077    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015080    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015083    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015086    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015088    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015091    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015093    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015096    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015098    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.017520 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015101    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015103    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015105    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015108    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015110    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015112    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015115    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015118    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015120    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015122    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015125    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015128    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015131    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015134    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015136    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015138    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015141    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015144    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015146    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015149    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.018019 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015151    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015153    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015156    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015159    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015163    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015166    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015170    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015173    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015176    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015178    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015181    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015183    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015185    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015188    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015190    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015193    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015195    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015198    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015200    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.018527 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015202    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015613    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015618    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015624    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015627    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015629    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015632    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015634    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015637    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015639    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015642    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015644    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015647    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015649    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015652    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015654    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015657    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015660    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015662    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015665    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.019024 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015668    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015671    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015673    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015676    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015680    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015683    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015686    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015690    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015693    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015696    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015699    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015701    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015704    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015706    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015709    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015714    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015718    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015721    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.019517 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015724    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015726    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015729    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015731    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015735    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015738    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015740    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015743    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015745    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015748    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015753    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015756    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015758    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015761    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015764    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015767    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015769    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015772    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015774    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015776    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.019961 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015779    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015781    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015784    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015786    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015789    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015791    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015793    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015796    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015799    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015803    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015806    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015808    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015810    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015813    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015816    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015819    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015821    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015823    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015826    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015828    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.020450 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015831    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015833    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015837    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015840    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015842    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015844    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015847    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015850    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015852    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015854    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015857    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015859    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015861    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015864    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015866    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015869    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015871    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015873    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015876    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015878    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.020955 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015881    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.015885    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.015980    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.015991    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016001    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016007    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016012    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016015    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016020    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016025    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016028    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016031    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016035    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016039    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016043    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016046    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016049    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016052    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016055    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016058    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016073    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016076    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016079    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016083    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:34:10.021444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016086    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016090    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016093    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016096    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016099    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016102    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016105    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016108    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016111    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016114    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016118    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016121    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016124    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016127    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016130    2559 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016132    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016137    2559 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016140    2559 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016143    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016146    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016149    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016152    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016157    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016160    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016163    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:34:10.022021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016165    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016168    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016171    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016173    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016177    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016180    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016183    2559 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016186    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016189    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016192    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016195    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016198    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016200    2559 flags.go:64] FLAG: --help="false"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016203    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016206    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016209    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016212    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016216    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016219    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016222    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016225    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016227    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016230    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:34:10.022682 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016233    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016236    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016239    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016241    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016244    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016247    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016252    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016254    2559 flags.go:64] FLAG: --lock-file=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016257    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016260    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016263    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016268    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016271    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016274    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016277    2559 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016280    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016283    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016286    2559 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016289    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016293    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016296    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016300    2559 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016303    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016306    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016308    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:34:10.023275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016312    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016315    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016318    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016321    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016329    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016332    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016335    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016338    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016341    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016347    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016349    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016352    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016355    2559 flags.go:64] FLAG: --port="10250"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016359    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016362    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0856ed9cdde7c6464"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016366    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016368    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016371    2559 flags.go:64] FLAG: --register-node="true"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016374    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016378    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016381    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016383    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016387    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016390    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016393    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:34:10.023901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016396    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016399    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016402    2559 flags.go:64] FLAG: --runonce="false"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016405    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016408    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016411    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016413    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016416    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016419    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016422    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016425    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016427    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016430    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016433    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016435    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016438    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016441    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016444    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016449    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016452    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016456    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016460    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016462    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016465    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016468    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:34:10.024645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016471    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016474    2559 flags.go:64] FLAG: --v="2"
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016477    2559 flags.go:64] FLAG: --version="false"
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016482    2559 flags.go:64] FLAG: --vmodule=""
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016485    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.016492    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016624    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016629    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016632    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016635    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016637    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016640    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016643    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016645    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016648    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016652    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016655    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016657    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016660    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016662    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016665    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016668    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.025655 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016670    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016673    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016675    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016678    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016680    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016683    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016686    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016688    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016690    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016693    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016697    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016700    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016703    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016706    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016709    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016712    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016717    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016719    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016722    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.026217 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016725    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016727    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016730    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016732    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016735    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016737    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016741    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016744    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016746    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016748    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016751    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016754    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016756    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016759    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016761    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016764    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016766    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016769    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016771    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016774    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.026763 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016776    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016779    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016781    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016784    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016786    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016788    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016791    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016793    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016796    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016798    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016801    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016804    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016806    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016808    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016811    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016813    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016816    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016818    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016822    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.027246 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016824    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016827    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016829    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016832    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016834    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016837    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016839    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016841    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016844    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016846    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016849    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016851    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016854    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016856    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016858    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016861    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016863    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016865    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016868    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.027761 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016870    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016874    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016878    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016880    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016883    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.016886    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.017415    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.025217    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.025244    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.025270    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.025279    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025440    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025448    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025453    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.028219 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025464    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025468    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025473    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025477    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025484    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025490    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025494    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025499    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025503    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025508    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025513    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025517    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025521    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025531    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025540    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025545    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025548    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025550    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025553    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025556    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.028584 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025559    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025562    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025566    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025586    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025591    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025596    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025603    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025627    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025630    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025634    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025744    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025748    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025750    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025753    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025756    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025759    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025761    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025766    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025770    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.029066 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025773    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025775    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025778    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025781    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025784    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025787    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025790    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025792    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025795    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025797    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025804    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025807    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025810    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025812    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025815    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025817    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025820    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025822    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025825    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025827    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.029529 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025830    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025833    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025837    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025840    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025843    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025846    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025849    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025852    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025856    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025859    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025862    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025865    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025868    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025870    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025872    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025875    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025878    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025881    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025883    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.030038 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025887    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025890    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025892    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025895    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025897    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025899    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025902    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025904    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025907    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025909    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025912    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025914    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025917    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025920    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025922    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025925    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025927    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.030524 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.025930    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.025936    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026054    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026060    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026063    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026066    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026069    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026072    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026075    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026078    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026080    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026083    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026086    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026088    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026091    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:10.031036 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026094    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026096    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026099    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026102    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026104    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026107    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026109    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026111    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026115    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026119    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026122    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026124    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026127    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026130    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026133    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026135    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026138    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026140    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026143    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:10.031408 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026146    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026148    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026152    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026154    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026157    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026159    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026161    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026164    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026167    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026169    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026171    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026174    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026176    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026179    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026182    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026184    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026187    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026190    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026193    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026195    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:10.031903 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026197    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026199    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026202    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026204    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026207    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026210    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026212    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026215    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026217    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026220    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026223    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026225    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026228    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026230    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026233    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026235    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026238    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026240    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026242    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026245    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:10.032402 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026247    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026249    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026251    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026254    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026256    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026259    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026262    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026264    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026267    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026269    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026272    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026275    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026278    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026281    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026284    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026286    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026289    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026291    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026294    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026296    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:10.032890 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026298    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026301    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026303    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026306    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026308    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026311    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:10.026314    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.026319    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.026435    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.029276    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.030602    2559 server.go:1019] "Starting client certificate rotation"
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.030702    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:10.033368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.030743    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:10.033685 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.033533    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:10.039310 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.039289    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:10.047461 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.047446    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:34:10.050759 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.050741    2559 log.go:25] "Validated CRI v1 image API"
Jul 08 16:34:10.051606 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.051590    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:34:10.053651 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.053625    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d07d0af3-202e-44b4-9e66-2c8798dace3c:/dev/nvme0n1p4 f8fd6c05-1063-42c2-8a96-771d6c9bb753:/dev/nvme0n1p3]
Jul 08 16:34:10.053696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.053651    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:34:10.058607 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.058477    2559 manager.go:216] Machine: {Timestamp:2026-07-08 16:34:10.05800264 +0000 UTC m=+0.276199139 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100906 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2926c997d8161c337df98d8a808367 SystemUUID:ec2926c9-97d8-161c-337d-f98d8a808367 BootID:6a9dad09-2792-4297-9863-ae4c66f76ebb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:30:70:aa:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:30:70:aa:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:7b:a6:59:52:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:34:10.058607 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.058605    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:34:10.058717 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.058705    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:34:10.060312 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.060288    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:34:10.060445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.060315    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-182.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:34:10.060960 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.060950    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:34:10.061000 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.060962    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:34:10.061000 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.060971    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:10.061524 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.061514    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:10.061777 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.061761    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:10.062841 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.062828    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:34:10.065489 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065478    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:10.065674 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065664    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:34:10.065709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065698    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:34:10.065737 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065716    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:34:10.065737 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065731    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:34:10.065795 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065741    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:34:10.065795 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.065754    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:34:10.066130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.066119    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:10.066188 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.066136    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:10.068784 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.068752    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:34:10.071208 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.071188    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:34:10.071290 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.071232    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:34:10.072436 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072424    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072441    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072448    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072454    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072462    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072468    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072476    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:34:10.072484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072482    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:34:10.072694 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072488    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:34:10.072694 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072494    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:34:10.072694 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072503    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:34:10.072694 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.072508    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:34:10.073566 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.073555    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:34:10.073566 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.073566    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:34:10.077207 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.077188    2559 server.go:1268] "Started kubelet"
Jul 08 16:34:10.077775 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.077731    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:34:10.077881 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.077771    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:34:10.077881 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.077767    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:34:10.077881 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.077813    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-182.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:34:10.077881 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.077815    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:34:10.077881 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.077842    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:34:10.078095 ip-10-0-130-182 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:34:10.079491 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.079471    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:34:10.079920 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.079903    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:34:10.082887 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.082862    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:34:10.084909 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.084887    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:10.085003 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.084888    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:34:10.085094 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.085074    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.085623 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085601    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:34:10.085623 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085612    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:34:10.085792 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085634    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:34:10.085792 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085703    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:34:10.085792 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085712    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:34:10.085932 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085903    2559 factory.go:55] Registering systemd factory
Jul 08 16:34:10.085932 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.085918    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:34:10.086156 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086135    2559 factory.go:153] Registering CRI-O factory
Jul 08 16:34:10.086156 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086158    2559 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:34:10.086304 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086213    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:34:10.086304 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086232    2559 factory.go:103] Registering Raw factory
Jul 08 16:34:10.086304 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086242    2559 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:34:10.086589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.086561    2559 manager.go:349] Starting recovery of all containers
Jul 08 16:34:10.086713 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.085869    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18c05dad01a6f7d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-08 16:34:10.07716143 +0000 UTC m=+0.295357929,LastTimestamp:2026-07-08 16:34:10.07716143 +0000 UTC m=+0.295357929,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}"
Jul 08 16:34:10.088829 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.088805    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:34:10.088925 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.088903    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:34:10.095006 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.094984    2559 manager.go:354] Recovery completed
Jul 08 16:34:10.102407 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.102390    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.104543 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.104526    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.104641 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.104557    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.104641 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.104583    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.105153 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105137    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:34:10.105153 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105151    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:34:10.105256 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105157    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.105256 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105171    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:10.105935 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105920    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.106009 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105948    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.106009 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.105959    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.106248 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.106216    2559 policy_none.go:49] "None policy: Start"
Jul 08 16:34:10.106280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.106253    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:34:10.106280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.106265    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:34:10.107199 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.107185    2559 policy_none.go:47] "Start"
Jul 08 16:34:10.107272 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.107162    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18c05dad0348c424  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-08 16:34:10.104542244 +0000 UTC m=+0.322738744,LastTimestamp:2026-07-08 16:34:10.104542244 +0000 UTC m=+0.322738744,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}"
Jul 08 16:34:10.114115 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.114095    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nq6lr"
Jul 08 16:34:10.121420 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.121401    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nq6lr"
Jul 08 16:34:10.122984 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.122917    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18c05dad03490fc9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-08 16:34:10.104561609 +0000 UTC m=+0.322758109,LastTimestamp:2026-07-08 16:34:10.104561609 +0000 UTC m=+0.322758109,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}"
Jul 08 16:34:10.141846 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.141834    2559 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:34:10.141935 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.141876    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:34:10.141935 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.141890    2559 server.go:87] "Starting device plugin registration server"
Jul 08 16:34:10.142188 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.142171    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:34:10.142231 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.142192    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:34:10.142332 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.142316    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:34:10.142833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.142403    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:34:10.142833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.142416    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:34:10.142982 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.142920    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:34:10.143033 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.142981    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.236776 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.236729    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:34:10.238182 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.238160    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:34:10.238265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.238212    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:34:10.238265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.238244    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:34:10.238362 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.238294    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:34:10.241006 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.240986    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:10.242326 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.242313    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.243117 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.243102    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.243176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.243136    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.243176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.243147    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.243176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.243170    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.251697 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.251681    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.251768 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.251704    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-182.ec2.internal\": node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.276356 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.276280    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.338860 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.338825    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"]
Jul 08 16:34:10.338938 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.338921    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.340029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.340014    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.340117 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.340046    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.340117 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.340061    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.341208 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341195    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.341325 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.341362 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341356    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.341929 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341904    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.342007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341938    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.342007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341953    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.342073 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.341937    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.342108 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.342089    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.342108 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.342106    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.342919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.342907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.342964 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.342931    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:10.343654 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.343637    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:10.343722 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.343665    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:10.343722 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.343677    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:10.367266 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.367243    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.371928 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.371908    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.376961 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.376945    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.387209 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.387187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.387318 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.387213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.387318 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.387229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.477485 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.477441    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.487847 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.487908 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.487908 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.487908 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487893    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.488015 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487915    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.488015 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.487920    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.578336 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.578246    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.670900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.670856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.675739 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:10.675715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:10.678352 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.678332    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.778958 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.778921    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.879515 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.879432    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:10.980081 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:10.980034    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:11.030484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.030455    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:34:11.030913 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.030615    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:11.080124 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:11.080098    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:11.123831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.123789    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:29:09 +0000 UTC" deadline="2028-03-18 11:11:19.162138274 +0000 UTC"
Jul 08 16:34:11.123831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.123826    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14850h37m8.038316393s"
Jul 08 16:34:11.181019 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:11.180983    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:11.258557 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.258527    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:11.259736 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:11.259707    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed7f726f53bfd3508e373fd03ee9ef0.slice/crio-e4a5c6b2740ed2281de9e220810322eb716d5c779cc8ac9fb95eac904408eb56 WatchSource:0}: Error finding container e4a5c6b2740ed2281de9e220810322eb716d5c779cc8ac9fb95eac904408eb56: Status 404 returned error can't find the container with id e4a5c6b2740ed2281de9e220810322eb716d5c779cc8ac9fb95eac904408eb56
Jul 08 16:34:11.259879 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.259865    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:11.259938 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.259895    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:11.259938 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.259908    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:11.260128 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:11.260107    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29248b30c20510f3cbc8d2bec2a059f1.slice/crio-2f0dcae334f5e04d2b256a7ea115083dc54c2746f5a8bfc2dc3bcdd0497bfbda WatchSource:0}: Error finding container 2f0dcae334f5e04d2b256a7ea115083dc54c2746f5a8bfc2dc3bcdd0497bfbda: Status 404 returned error can't find the container with id 2f0dcae334f5e04d2b256a7ea115083dc54c2746f5a8bfc2dc3bcdd0497bfbda
Jul 08 16:34:11.264214 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.264192    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:34:11.281758 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:11.281732    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:11.382308 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:11.382274    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found"
Jul 08 16:34:11.465212 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.465137    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:11.486054 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.486032    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:11.499887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.499859    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:11.501289 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.501274    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"
Jul 08 16:34:11.510325 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.510312    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:11.613733 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.613708    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:11.637896 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:11.637874    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:12.067267 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.067226    2559 apiserver.go:52] "Watching apiserver"
Jul 08 16:34:12.079596 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.079394    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:34:12.080005 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.079981    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-v4744","kube-system/konnectivity-agent-bbnng","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal","openshift-dns/node-resolver-d558v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","openshift-multus/multus-8pv6c","openshift-multus/multus-additional-cni-plugins-stzpv","openshift-network-diagnostics/network-check-target-zsdxr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr","openshift-cluster-node-tuning-operator/tuned-4m96v","openshift-image-registry/node-ca-mdplr","openshift-multus/network-metrics-daemon-nb49f","openshift-network-operator/iptables-alerter-p9f7p","openshift-ovn-kubernetes/ovnkube-node-rqpfq"]
Jul 08 16:34:12.082670 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.082648    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.084749 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.084389    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.085774 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.085597    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:34:12.085774 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.085752    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.085951 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.085756    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.085951 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.085831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cbrbd\""
Jul 08 16:34:12.086050 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.085961    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:12.086050 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.086040    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:34:12.086538 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.086223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.086818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.087114    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.087438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmkgx\""
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.087843    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.088267    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.090592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.089459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.091231 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091205    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.091231 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091224    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:34:12.091365 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:34:12.091365 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:34:12.091458 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9tkdh\""
Jul 08 16:34:12.091852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.091625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.091852 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.091696    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:12.092917 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.092621    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.092917 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.092735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:34:12.093162 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.093147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.094184 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.093872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.094548 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.094517    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r75cz\""
Jul 08 16:34:12.094638 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.094552    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.095886 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.095870    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:34:12.096129 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.096438 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096413    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-ovn\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096522 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096522 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096483    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovn-node-metrics-cert\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096522 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-system-cni-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.096698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096554    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.096698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-systemd-units\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-systemd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-var-lib-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-script-lib\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nvm\" (UniqueName: \"kubernetes.io/projected/6d2a3089-f878-4543-8cec-c7dc83e9c457-kube-api-access-l6nvm\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096766    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-host-slash\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-bin\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-config\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.096862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-netd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmth8\" (UniqueName: \"kubernetes.io/projected/81401226-bc48-426d-878e-bca23eb723e4-kube-api-access-mmth8\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-netns\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-log-socket\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.096980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-node-log\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-cnibin\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-os-release\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-kubelet\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-slash\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-etc-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-env-overrides\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2lf\" (UniqueName: \"kubernetes.io/projected/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-kube-api-access-js2lf\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-iptables-alerter-script\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.097425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.097784 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jx6j6\""
Jul 08 16:34:12.097879 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.097863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.098992 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.098199    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.098992 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.098490    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rcx72\""
Jul 08 16:34:12.098992 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.098512    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.098992 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.098684    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:34:12.098992 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.098750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.099817 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.099668    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:12.100585 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.100556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.101643 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.101613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:34:12.103133 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.103021    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:34:12.103133 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.103032    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:34:12.103133 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.103032    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7czv\""
Jul 08 16:34:12.106013 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.105995    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.108486 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.108470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.108596 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.108542    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:12.109184 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.109164    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-979db\""
Jul 08 16:34:12.109432 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.109413    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:12.109528 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.109512    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:12.110394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.110379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:12.110457 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.110439    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:12.123291 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.121903    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vpbr"
Jul 08 16:34:12.124514 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.124489    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f2wl6\""
Jul 08 16:34:12.138787 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.138761    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vpbr"
Jul 08 16:34:12.186689 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.186658    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:12.186921 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.186905    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:34:12.197932 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.197901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-cnibin\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.197942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-os-release\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.197968    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-etc-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.197985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-env-overrides\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2lf\" (UniqueName: \"kubernetes.io/projected/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-kube-api-access-js2lf\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-lib-modules\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tj7\" (UniqueName: \"kubernetes.io/projected/26860848-e85c-4136-9004-bb2c83bfe483-kube-api-access-c5tj7\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-host\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.198130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-k8s-cni-cncf-io\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-conf-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-iptables-alerter-script\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-ovn\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovn-node-metrics-cert\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-bin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-kubelet\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-conf\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198349    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-var-lib-kubelet\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nvm\" (UniqueName: \"kubernetes.io/projected/6d2a3089-f878-4543-8cec-c7dc83e9c457-kube-api-access-l6nvm\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198430 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7hp\" (UniqueName: \"kubernetes.io/projected/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-kube-api-access-qq7hp\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj6db\" (UniqueName: \"kubernetes.io/projected/eeb425bc-0259-4c32-af34-8b5a977324fa-kube-api-access-qj6db\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198472    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-cni-binary-copy\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-config\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec01533-0ac2-429d-aa0d-b016eed8f657-agent-certs\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-modprobe-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-host\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-run\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9d99\" (UniqueName: \"kubernetes.io/projected/de19f666-e1a6-4638-a51c-c7a969ff69d0-kube-api-access-m9d99\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmth8\" (UniqueName: \"kubernetes.io/projected/81401226-bc48-426d-878e-bca23eb723e4-kube-api-access-mmth8\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-multus-certs\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-kubelet\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.198862 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-slash\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25359783-c9a1-4d17-b19d-cd028d55d434-hosts-file\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-os-release\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-socket-dir-parent\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-hostroot\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.198990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199051    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-registration-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-kubernetes\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-systemd\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-kubelet-config\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-system-cni-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-systemd-units\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-systemd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-var-lib-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-script-lib\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec01533-0ac2-429d-aa0d-b016eed8f657-konnectivity-ca\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-host-slash\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199413    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-bin\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-dbus\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-serviceca\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-etc-kubernetes\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199587    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-netd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-sys\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-tuned\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-tmp\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25359783-c9a1-4d17-b19d-cd028d55d434-tmp-dir\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9l8s\" (UniqueName: \"kubernetes.io/projected/25359783-c9a1-4d17-b19d-cd028d55d434-kube-api-access-z9l8s\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.199799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-cnibin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-netns\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-log-socket\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysconfig\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-device-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-sys-fs\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-netns\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-multus-daemon-config\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-node-log\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.199980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-socket-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-system-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-multus\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfp2t\" (UniqueName: \"kubernetes.io/projected/7f231f0b-d60d-4606-a528-5c77859ace84-kube-api-access-tfp2t\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200160    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-cnibin\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200234    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-os-release\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.200280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200275    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-etc-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.200833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.200814    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-env-overrides\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.201481 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.201462    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-iptables-alerter-script\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.201598 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.201517    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-ovn\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.201598 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.201558    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.201825 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.201798    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:34:12.202082 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.201994    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.202165 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202118    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-kubelet\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.202165 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202160    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-slash\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.202755 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202735    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.202833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202797    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.202833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202739    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.202907 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.202854    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203236 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203215    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-config\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203321 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203278    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-netd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203366 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203326    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-var-lib-openvswitch\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203366 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203329    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-log-socket\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203366 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-run-systemd\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203490 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203379    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-systemd-units\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203490 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203397    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-run-netns\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203490 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203417    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81401226-bc48-426d-878e-bca23eb723e4-system-cni-dir\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.203790 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203771    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/81401226-bc48-426d-878e-bca23eb723e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.203790 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203786    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovnkube-script-lib\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203867 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203820    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-node-log\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.203867 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203823    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-host-slash\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.203867 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.203861    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d2a3089-f878-4543-8cec-c7dc83e9c457-host-cni-bin\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.205351 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.205307    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d2a3089-f878-4543-8cec-c7dc83e9c457-ovn-node-metrics-cert\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.219008 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.218597    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmth8\" (UniqueName: \"kubernetes.io/projected/81401226-bc48-426d-878e-bca23eb723e4-kube-api-access-mmth8\") pod \"multus-additional-cni-plugins-stzpv\" (UID: \"81401226-bc48-426d-878e-bca23eb723e4\") " pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.219008 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.218720    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nvm\" (UniqueName: \"kubernetes.io/projected/6d2a3089-f878-4543-8cec-c7dc83e9c457-kube-api-access-l6nvm\") pod \"ovnkube-node-rqpfq\" (UID: \"6d2a3089-f878-4543-8cec-c7dc83e9c457\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.219008 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.218971    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2lf\" (UniqueName: \"kubernetes.io/projected/2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2-kube-api-access-js2lf\") pod \"iptables-alerter-p9f7p\" (UID: \"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2\") " pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.243953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.243901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"bed7f726f53bfd3508e373fd03ee9ef0","Type":"ContainerStarted","Data":"e4a5c6b2740ed2281de9e220810322eb716d5c779cc8ac9fb95eac904408eb56"}
Jul 08 16:34:12.245111 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.245076    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerStarted","Data":"2f0dcae334f5e04d2b256a7ea115083dc54c2746f5a8bfc2dc3bcdd0497bfbda"}
Jul 08 16:34:12.300549 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-conf\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-var-lib-kubelet\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7hp\" (UniqueName: \"kubernetes.io/projected/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-kube-api-access-qq7hp\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj6db\" (UniqueName: \"kubernetes.io/projected/eeb425bc-0259-4c32-af34-8b5a977324fa-kube-api-access-qj6db\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-cni-binary-copy\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec01533-0ac2-429d-aa0d-b016eed8f657-agent-certs\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300694    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-conf\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-modprobe-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-host\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-run\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300784    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-modprobe-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300816    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-host\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.300945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300842    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-run\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301097 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300963    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-var-lib-kubelet\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301292 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.300785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9d99\" (UniqueName: \"kubernetes.io/projected/de19f666-e1a6-4638-a51c-c7a969ff69d0-kube-api-access-m9d99\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301326 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.301361 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-multus-certs\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301361 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301346    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-cni-binary-copy\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301441 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.301441 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301388    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.301441 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.301412    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:12.301605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301458    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-multus-certs\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301605 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.301501    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:12.80146819 +0000 UTC m=+3.019664691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:12.301605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25359783-c9a1-4d17-b19d-cd028d55d434-hosts-file\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.301605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-os-release\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-socket-dir-parent\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301604    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25359783-c9a1-4d17-b19d-cd028d55d434-hosts-file\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-hostroot\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301628    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-os-release\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301659    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-socket-dir-parent\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-registration-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301672    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-hostroot\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-kubernetes\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301703    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-registration-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-systemd\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-kubelet-config\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec01533-0ac2-429d-aa0d-b016eed8f657-konnectivity-ca\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301787    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-kubelet-config\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-dbus\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301742    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-kubernetes\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301774    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-systemd\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.301831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-serviceca\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-etc-kubernetes\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-sys\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-tuned\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-tmp\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25359783-c9a1-4d17-b19d-cd028d55d434-tmp-dir\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301993    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.301916    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-dbus\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9l8s\" (UniqueName: \"kubernetes.io/projected/25359783-c9a1-4d17-b19d-cd028d55d434-kube-api-access-z9l8s\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-cnibin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302035    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-etc-kubernetes\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysconfig\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-device-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302077    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-sys\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302128    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-cnibin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302163    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysconfig\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.302406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302202    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-device-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302236    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-serviceca\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302238    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5ec01533-0ac2-429d-aa0d-b016eed8f657-konnectivity-ca\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.302317    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.302355    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:12.802341686 +0000 UTC m=+3.020538172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302383    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-sys-fs\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302407    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-netns\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-multus-daemon-config\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302455    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25359783-c9a1-4d17-b19d-cd028d55d434-tmp-dir\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-socket-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-system-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-multus\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfp2t\" (UniqueName: \"kubernetes.io/projected/7f231f0b-d60d-4606-a528-5c77859ace84-kube-api-access-tfp2t\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-lib-modules\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302613    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-sysctl-d\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tj7\" (UniqueName: \"kubernetes.io/projected/26860848-e85c-4136-9004-bb2c83bfe483-kube-api-access-c5tj7\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.303118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-host\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-k8s-cni-cncf-io\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-conf-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-bin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-kubelet\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302865    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-kubelet\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302887    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f231f0b-d60d-4606-a528-5c77859ace84-multus-daemon-config\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302891    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-socket-dir\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302905    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-host\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302928    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-netns\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302947    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-multus\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302952    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eeb425bc-0259-4c32-af34-8b5a977324fa-sys-fs\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.302977    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-run-k8s-cni-cncf-io\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303020    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-system-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303052    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-conf-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303053    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de19f666-e1a6-4638-a51c-c7a969ff69d0-lib-modules\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303086    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-host-var-lib-cni-bin\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.303906 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303136    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f231f0b-d60d-4606-a528-5c77859ace84-multus-cni-dir\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.304686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.303897    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5ec01533-0ac2-429d-aa0d-b016eed8f657-agent-certs\") pod \"konnectivity-agent-bbnng\" (UID: \"5ec01533-0ac2-429d-aa0d-b016eed8f657\") " pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.304686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.304441    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-tmp\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.304885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.304868    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de19f666-e1a6-4638-a51c-c7a969ff69d0-etc-tuned\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.317064 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.316725    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:12.317064 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.316784    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:12.317064 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.316809    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:12.317064 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.316884    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:12.816865107 +0000 UTC m=+3.035061608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:12.318425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.318370    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj6db\" (UniqueName: \"kubernetes.io/projected/eeb425bc-0259-4c32-af34-8b5a977324fa-kube-api-access-qj6db\") pod \"aws-ebs-csi-driver-node-p47tr\" (UID: \"eeb425bc-0259-4c32-af34-8b5a977324fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.320806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.320781    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9d99\" (UniqueName: \"kubernetes.io/projected/de19f666-e1a6-4638-a51c-c7a969ff69d0-kube-api-access-m9d99\") pod \"tuned-4m96v\" (UID: \"de19f666-e1a6-4638-a51c-c7a969ff69d0\") " pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.321201 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.321174    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7hp\" (UniqueName: \"kubernetes.io/projected/797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1-kube-api-access-qq7hp\") pod \"node-ca-mdplr\" (UID: \"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1\") " pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.321476 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.321455    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9l8s\" (UniqueName: \"kubernetes.io/projected/25359783-c9a1-4d17-b19d-cd028d55d434-kube-api-access-z9l8s\") pod \"node-resolver-d558v\" (UID: \"25359783-c9a1-4d17-b19d-cd028d55d434\") " pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.321528 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.321496    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tj7\" (UniqueName: \"kubernetes.io/projected/26860848-e85c-4136-9004-bb2c83bfe483-kube-api-access-c5tj7\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.321621 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.321599    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfp2t\" (UniqueName: \"kubernetes.io/projected/7f231f0b-d60d-4606-a528-5c77859ace84-kube-api-access-tfp2t\") pod \"multus-8pv6c\" (UID: \"7f231f0b-d60d-4606-a528-5c77859ace84\") " pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.398608 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.398557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stzpv"
Jul 08 16:34:12.406103 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.406079    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9f7p"
Jul 08 16:34:12.415725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.415702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:12.424242 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.424221    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d558v"
Jul 08 16:34:12.432775 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.432755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mdplr"
Jul 08 16:34:12.438255 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.438228    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr"
Jul 08 16:34:12.451881 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.451857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8pv6c"
Jul 08 16:34:12.457449 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.457432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:12.464055 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.464031    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4m96v"
Jul 08 16:34:12.805557 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.805478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:12.805557 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.805536    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:12.805776 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.805616    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:12.805776 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.805668    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:13.805650297 +0000 UTC m=+4.023846794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:12.805776 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.805694    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:12.805776 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.805761    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:13.805744905 +0000 UTC m=+4.023941396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:12.906978 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:12.906377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:12.906978 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.906527    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:12.906978 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.906559    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:12.906978 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.906590    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:12.906978 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:12.906650    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:13.906633955 +0000 UTC m=+4.124830453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:13.142176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.142148    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:11 +0000 UTC" deadline="2028-03-13 20:55:10.109278135 +0000 UTC"
Jul 08 16:34:13.142176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.142176    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14740h20m56.967105774s"
Jul 08 16:34:13.158711 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.158682    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2a3089_f878_4543_8cec_c7dc83e9c457.slice/crio-761489d9f2c0dc3c074ef254c06fdf2299a59b5a93519641e0c6538e404cd2b1 WatchSource:0}: Error finding container 761489d9f2c0dc3c074ef254c06fdf2299a59b5a93519641e0c6538e404cd2b1: Status 404 returned error can't find the container with id 761489d9f2c0dc3c074ef254c06fdf2299a59b5a93519641e0c6538e404cd2b1
Jul 08 16:34:13.165082 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.165058    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81401226_bc48_426d_878e_bca23eb723e4.slice/crio-fa54c80e3bdae9eca285dec40837c1767273c82dcffc00846cbd3359c5944a36 WatchSource:0}: Error finding container fa54c80e3bdae9eca285dec40837c1767273c82dcffc00846cbd3359c5944a36: Status 404 returned error can't find the container with id fa54c80e3bdae9eca285dec40837c1767273c82dcffc00846cbd3359c5944a36
Jul 08 16:34:13.183297 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.183272    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb425bc_0259_4c32_af34_8b5a977324fa.slice/crio-5ce021aef273a80a83c4518d67ee458ef1646d56839441ecc5b01f4a3a539a8e WatchSource:0}: Error finding container 5ce021aef273a80a83c4518d67ee458ef1646d56839441ecc5b01f4a3a539a8e: Status 404 returned error can't find the container with id 5ce021aef273a80a83c4518d67ee458ef1646d56839441ecc5b01f4a3a539a8e
Jul 08 16:34:13.183907 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.183884    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25359783_c9a1_4d17_b19d_cd028d55d434.slice/crio-209fe58df6c08d2bf630ecb873b2a296631bf012c402134d556c1ad14096e2bf WatchSource:0}: Error finding container 209fe58df6c08d2bf630ecb873b2a296631bf012c402134d556c1ad14096e2bf: Status 404 returned error can't find the container with id 209fe58df6c08d2bf630ecb873b2a296631bf012c402134d556c1ad14096e2bf
Jul 08 16:34:13.184776 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.184723    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec01533_0ac2_429d_aa0d_b016eed8f657.slice/crio-350459da1d1533a4858603b50eb2320f9c3149d1b79b9f3d33f258c7f0e6af58 WatchSource:0}: Error finding container 350459da1d1533a4858603b50eb2320f9c3149d1b79b9f3d33f258c7f0e6af58: Status 404 returned error can't find the container with id 350459da1d1533a4858603b50eb2320f9c3149d1b79b9f3d33f258c7f0e6af58
Jul 08 16:34:13.185406 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.185383    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde19f666_e1a6_4638_a51c_c7a969ff69d0.slice/crio-eb9a587d25be770457cebfebebf17bbba4094a0f7e95c47a8e64fb1decb3548a WatchSource:0}: Error finding container eb9a587d25be770457cebfebebf17bbba4094a0f7e95c47a8e64fb1decb3548a: Status 404 returned error can't find the container with id eb9a587d25be770457cebfebebf17bbba4094a0f7e95c47a8e64fb1decb3548a
Jul 08 16:34:13.186751 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.186458    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaa4ca0_dbdb_4199_9ff9_dfc32a9868b2.slice/crio-007571af7e3f3b55fbe3e3421e0f4c39821e2f83adc684eebcc64d2f6a82172f WatchSource:0}: Error finding container 007571af7e3f3b55fbe3e3421e0f4c39821e2f83adc684eebcc64d2f6a82172f: Status 404 returned error can't find the container with id 007571af7e3f3b55fbe3e3421e0f4c39821e2f83adc684eebcc64d2f6a82172f
Jul 08 16:34:13.188129 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.188080    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797e5d6b_c7bb_4df7_a1c8_297eca0fa2e1.slice/crio-d990ca72ed197e1c1b99414176d60de61337adbd687f2a919bd5e7a1b0cfc07d WatchSource:0}: Error finding container d990ca72ed197e1c1b99414176d60de61337adbd687f2a919bd5e7a1b0cfc07d: Status 404 returned error can't find the container with id d990ca72ed197e1c1b99414176d60de61337adbd687f2a919bd5e7a1b0cfc07d
Jul 08 16:34:13.188500 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:13.188471    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f231f0b_d60d_4606_a528_5c77859ace84.slice/crio-df26be67f0e07cd24b46a72e1fa78b4c6bced1c0f7a2fb5f4262ad183f460155 WatchSource:0}: Error finding container df26be67f0e07cd24b46a72e1fa78b4c6bced1c0f7a2fb5f4262ad183f460155: Status 404 returned error can't find the container with id df26be67f0e07cd24b46a72e1fa78b4c6bced1c0f7a2fb5f4262ad183f460155
Jul 08 16:34:13.239114 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.238960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:13.239242 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.239001    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:13.239242 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.239221    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:13.239317 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.239254    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:13.247860 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.247837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8pv6c" event={"ID":"7f231f0b-d60d-4606-a528-5c77859ace84","Type":"ContainerStarted","Data":"df26be67f0e07cd24b46a72e1fa78b4c6bced1c0f7a2fb5f4262ad183f460155"}
Jul 08 16:34:13.249442 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.249418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mdplr" event={"ID":"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1","Type":"ContainerStarted","Data":"d990ca72ed197e1c1b99414176d60de61337adbd687f2a919bd5e7a1b0cfc07d"}
Jul 08 16:34:13.252272 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.252250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4m96v" event={"ID":"de19f666-e1a6-4638-a51c-c7a969ff69d0","Type":"ContainerStarted","Data":"eb9a587d25be770457cebfebebf17bbba4094a0f7e95c47a8e64fb1decb3548a"}
Jul 08 16:34:13.253244 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.253221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d558v" event={"ID":"25359783-c9a1-4d17-b19d-cd028d55d434","Type":"ContainerStarted","Data":"209fe58df6c08d2bf630ecb873b2a296631bf012c402134d556c1ad14096e2bf"}
Jul 08 16:34:13.254299 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.254279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr" event={"ID":"eeb425bc-0259-4c32-af34-8b5a977324fa","Type":"ContainerStarted","Data":"5ce021aef273a80a83c4518d67ee458ef1646d56839441ecc5b01f4a3a539a8e"}
Jul 08 16:34:13.255279 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.255258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerStarted","Data":"fa54c80e3bdae9eca285dec40837c1767273c82dcffc00846cbd3359c5944a36"}
Jul 08 16:34:13.256333 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.256315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"761489d9f2c0dc3c074ef254c06fdf2299a59b5a93519641e0c6538e404cd2b1"}
Jul 08 16:34:13.257354 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.257332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9f7p" event={"ID":"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2","Type":"ContainerStarted","Data":"007571af7e3f3b55fbe3e3421e0f4c39821e2f83adc684eebcc64d2f6a82172f"}
Jul 08 16:34:13.258352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.258332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bbnng" event={"ID":"5ec01533-0ac2-429d-aa0d-b016eed8f657","Type":"ContainerStarted","Data":"350459da1d1533a4858603b50eb2320f9c3149d1b79b9f3d33f258c7f0e6af58"}
Jul 08 16:34:13.814294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.814211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:13.814294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.814287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:13.814498 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.814438    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:13.814498 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.814495    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:15.814478747 +0000 UTC m=+6.032675234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:13.814933 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.814891    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:13.814997 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.814944    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:15.81492822 +0000 UTC m=+6.033124712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:13.915507 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:13.915467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:13.915786 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.915701    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:13.915786 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.915724    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:13.915786 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.915735    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:13.915949 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:13.915792    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:15.915774372 +0000 UTC m=+6.133970860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:14.142665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.142563    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:11 +0000 UTC" deadline="2028-04-12 11:35:34.635015878 +0000 UTC"
Jul 08 16:34:14.142665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.142612    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15451h1m20.492408627s"
Jul 08 16:34:14.241665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.241130    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:14.241665 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:14.241251    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:14.278525 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.278483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"bed7f726f53bfd3508e373fd03ee9ef0","Type":"ContainerStarted","Data":"40ed6a9c58cd4fd3169512fa5663917da8f0e3cb10700088c1c7265836503329"}
Jul 08 16:34:14.291595 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.291318    2559 generic.go:356] "Generic (PLEG): container finished" podID="29248b30c20510f3cbc8d2bec2a059f1" containerID="560e829509edde52d6ba570f49b185b038abd4d5bf5da19c0fa1167eb87c9a25" exitCode=0
Jul 08 16:34:14.291595 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.291374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerDied","Data":"560e829509edde52d6ba570f49b185b038abd4d5bf5da19c0fa1167eb87c9a25"}
Jul 08 16:34:14.321624 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:14.321545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" podStartSLOduration=3.321525061 podStartE2EDuration="3.321525061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:14.298776359 +0000 UTC m=+4.516972866" watchObservedRunningTime="2026-07-08 16:34:14.321525061 +0000 UTC m=+4.539721568"
Jul 08 16:34:15.239477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.239445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:15.239947 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.239611    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:15.240014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.239986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:15.240093 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.240073    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:15.308384 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.308348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerStarted","Data":"faab417c68933364f0a7fd7e5958652717520c3487ccc10d7b0282bc41d0c7a0"}
Jul 08 16:34:15.328731 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.328674    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" podStartSLOduration=4.328653021 podStartE2EDuration="4.328653021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:15.3282582 +0000 UTC m=+5.546454705" watchObservedRunningTime="2026-07-08 16:34:15.328653021 +0000 UTC m=+5.546849532"
Jul 08 16:34:15.831776 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.831733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:15.832074 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.831823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:15.832074 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.831961    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:15.832074 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.832027    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:19.832007015 +0000 UTC m=+10.050203525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:15.832461 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.832438    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:15.832539 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.832501    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:19.832484754 +0000 UTC m=+10.050681244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:15.933170 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:15.933115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:15.933414 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.933383    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:15.933414 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.933411    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:15.933585 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.933424    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:15.933585 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:15.933486    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:19.933465659 +0000 UTC m=+10.151662163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:16.241902 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:16.241280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:16.241902 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:16.241412    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:17.238640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:17.238595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:17.238852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:17.238602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:17.238852 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:17.238750    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:17.238852 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:17.238769    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:18.239207 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:18.239169    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:18.239717 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:18.239311    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:19.238915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:19.238877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:19.238915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:19.238904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:19.239167 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.238999    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:19.239167 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.239126    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:19.870617 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:19.870554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:19.871109 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:19.870683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:19.871109 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.870814    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:19.871109 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.870865    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.870851908 +0000 UTC m=+18.089048395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:19.871280 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.871211    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:19.871280 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.871252    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.871239563 +0000 UTC m=+18.089436049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:19.971396 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:19.971351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:19.971611 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.971545    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:19.971611 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.971567    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:19.971684 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.971633    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:19.971760 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:19.971739    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.971720434 +0000 UTC m=+18.189916925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:20.240060 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:20.239985    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:20.240212 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:20.240101    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:21.239346 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:21.239264    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:21.239346 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:21.239305    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:21.239895 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:21.239401    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:21.239895 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:21.239590    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:22.238616 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:22.238553    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:22.238823 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:22.238708    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:23.238807 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:23.238768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:23.239213 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:23.238900    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:23.239213 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:23.238953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:23.239213 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:23.239070    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:24.238632 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:24.238509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:24.238800 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:24.238682    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:25.238972 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:25.238937    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:25.239444 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:25.239050    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:25.239444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:25.239110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:25.239444 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:25.239232    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:26.239619 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:26.239567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:26.240079 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:26.239727    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:27.238680 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:27.238637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:27.238922 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:27.238637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:27.238922 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.238779    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:27.238922 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.238909    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:27.931851 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:27.931813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:27.932385 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:27.931871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:27.932385 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.931967    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:27.932385 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.931974    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:27.932385 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.932020    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:43.932006562 +0000 UTC m=+34.150203050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:27.932385 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:27.932035    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:43.932027888 +0000 UTC m=+34.150224375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:28.032341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:28.032300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:28.032525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:28.032472    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:28.032525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:28.032491    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:28.032525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:28.032501    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:28.032677 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:28.032556    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:34:44.032542811 +0000 UTC m=+34.250739318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:28.239104 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:28.239024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:28.239247 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:28.239141    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:29.238923 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:29.238888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:29.239454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:29.238944    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:29.239454 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:29.239035    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:29.239454 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:29.239197    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:30.239291 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:30.239255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:30.239777 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:30.239367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:31.239439 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:31.239409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:31.239948 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:31.239409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:31.239948 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:31.239561    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:31.239948 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:31.239628    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:32.240806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:32.240778    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:32.241192 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:32.240903    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:33.238850 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.238501    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:33.239021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.238562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:33.239021 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:33.238972    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:33.239021 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:33.239002    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:33.354169 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.354131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d558v" event={"ID":"25359783-c9a1-4d17-b19d-cd028d55d434","Type":"ContainerStarted","Data":"bea6853dce09e986d95249e5f576427fd8cf023bc33a0683c66f4da32d4761ed"}
Jul 08 16:34:33.355890 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.355849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr" event={"ID":"eeb425bc-0259-4c32-af34-8b5a977324fa","Type":"ContainerStarted","Data":"f0236c14de5004e9ab0394730647b61f513af7914fc4f04ca771d720125556be"}
Jul 08 16:34:33.360427 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.360396    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="ecfa0cd01d5e58fe36b634b6cd235a4b0531d20f4e9efa68fa78b217d1811c8c" exitCode=0
Jul 08 16:34:33.360557 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.360489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"ecfa0cd01d5e58fe36b634b6cd235a4b0531d20f4e9efa68fa78b217d1811c8c"}
Jul 08 16:34:33.364205 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"ed8a37d147a11a5d34c8c2c1d85a76bb2f44eeed067fd87f85584f9b0b15d0d9"}
Jul 08 16:34:33.364307 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"bc4f1d2f25be9fc60481b7320ac2f909c7024847055d7bd44ee99e8627bbe440"}
Jul 08 16:34:33.364307 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"e95fac72ebd76726f06898b30d0fef92890b241987c608aaa0c5af8222a1433b"}
Jul 08 16:34:33.364444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"841245ead4756d6c1ccd44d001374f09032f44bd86c5e55d9ff05cf25f02beff"}
Jul 08 16:34:33.364444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"7058eb2fcff9e19cdd1b5a2e165f95821daa2680679ed98950b38e7afc3cea8e"}
Jul 08 16:34:33.364444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.364435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"705662bb9e49d3217102b8ef5fcad866884921664802d174cd3659d425a4061d"}
Jul 08 16:34:33.365818 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.365791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bbnng" event={"ID":"5ec01533-0ac2-429d-aa0d-b016eed8f657","Type":"ContainerStarted","Data":"3c512a9429f42b3bbbbcf46fb799cb61db04073fcc6e9f0ef7784b6a01f24cf1"}
Jul 08 16:34:33.367785 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.367762    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8pv6c" event={"ID":"7f231f0b-d60d-4606-a528-5c77859ace84","Type":"ContainerStarted","Data":"d80d6c93eb87dcd4a12c9789498f070217f567d4ae3331bf1f877cff1e76bf1a"}
Jul 08 16:34:33.369339 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.369315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mdplr" event={"ID":"797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1","Type":"ContainerStarted","Data":"af0e10caebe486044eee9cc06159b443dc9e6686dc039f763b0b9afd897d00aa"}
Jul 08 16:34:33.370767 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.370736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4m96v" event={"ID":"de19f666-e1a6-4638-a51c-c7a969ff69d0","Type":"ContainerStarted","Data":"068008df3c76f707eb31a769022902a4e9f727dc31ed421784c3293dfb064101"}
Jul 08 16:34:33.396858 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.396801    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d558v" podStartSLOduration=5.338924199 podStartE2EDuration="24.396778192s" totalImagesPullingTime="19.057853993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:33.372039167 +0000 UTC m=+23.590235676" watchObservedRunningTime="2026-07-08 16:34:33.396778192 +0000 UTC m=+23.614974701"
Jul 08 16:34:33.397047 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.396928    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8pv6c" podStartSLOduration=4.2613407930000005 podStartE2EDuration="23.396920699s" totalImagesPullingTime="19.135579906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:33.396204916 +0000 UTC m=+23.614401426" watchObservedRunningTime="2026-07-08 16:34:33.396920699 +0000 UTC m=+23.615117209"
Jul 08 16:34:33.417619 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.417530    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bbnng" podStartSLOduration=4.372662525 podStartE2EDuration="23.41750802s" totalImagesPullingTime="19.044845495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:33.417222483 +0000 UTC m=+23.635418991" watchObservedRunningTime="2026-07-08 16:34:33.41750802 +0000 UTC m=+23.635704532"
Jul 08 16:34:33.442477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.442419    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4m96v" podStartSLOduration=4.35996644 podStartE2EDuration="23.442401673s" totalImagesPullingTime="19.082435233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:33.442143346 +0000 UTC m=+23.660339855" watchObservedRunningTime="2026-07-08 16:34:33.442401673 +0000 UTC m=+23.660598173"
Jul 08 16:34:33.519098 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.519043    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mdplr" podStartSLOduration=4.47300866 podStartE2EDuration="23.519025703s" totalImagesPullingTime="19.046017043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:33.518677939 +0000 UTC m=+23.736874458" watchObservedRunningTime="2026-07-08 16:34:33.519025703 +0000 UTC m=+23.737222211"
Jul 08 16:34:33.644479 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:33.644451    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:34:34.154818 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.154676    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:34:33.644476866Z","UUID":"2eb63030-ab48-4d83-ad79-998eef6f5630","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:34:34.156405 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.156375    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:34:34.156552 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.156424    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:34:34.238882 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.238850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:34.239075 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:34.238972    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:34.374698 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.374661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9f7p" event={"ID":"2aaa4ca0-dbdb-4199-9ff9-dfc32a9868b2","Type":"ContainerStarted","Data":"e156d6f562d58f522d9e5fa6b1ee19ae0c4d5138d429ad875a0bed459af28dda"}
Jul 08 16:34:34.376514 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:34.376491    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr" event={"ID":"eeb425bc-0259-4c32-af34-8b5a977324fa","Type":"ContainerStarted","Data":"550ec7c0947a0785d00fd777740754b84659169b6001fd07236f4795e301ea14"}
Jul 08 16:34:35.239500 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.239460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:35.239700 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.239460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:35.239700 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:35.239641    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:35.239807 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:35.239723    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:35.382079 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.381978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr" event={"ID":"eeb425bc-0259-4c32-af34-8b5a977324fa","Type":"ContainerStarted","Data":"0e3feed0a885dd537c48311143b7b7a67da98959fe2a8ccb2c17c38cde08ab04"}
Jul 08 16:34:35.386374 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.386338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"a9787b64b3098f7bc89d64a92542f2375962e12c651ed064b0d078a1fae613f1"}
Jul 08 16:34:35.408413 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.408360    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p47tr" podStartSLOduration=4.11354411 podStartE2EDuration="25.40834629s" totalImagesPullingTime="21.29480218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:35.407351584 +0000 UTC m=+25.625548092" watchObservedRunningTime="2026-07-08 16:34:35.40834629 +0000 UTC m=+25.626542799"
Jul 08 16:34:35.408617 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:35.408564    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p9f7p" podStartSLOduration=7.367813975 podStartE2EDuration="26.408559965s" totalImagesPullingTime="19.04074599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:34.394676446 +0000 UTC m=+24.612872955" watchObservedRunningTime="2026-07-08 16:34:35.408559965 +0000 UTC m=+25.626756474"
Jul 08 16:34:36.238833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:36.238797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:36.239038 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:36.238908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:36.973587 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:36.973407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:36.974153 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:36.974134    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:37.238395 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.238375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:37.238459 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.238390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:37.238501 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:37.238485    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:37.238715 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:37.238688    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:37.394819 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.394775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" event={"ID":"6d2a3089-f878-4543-8cec-c7dc83e9c457","Type":"ContainerStarted","Data":"c8185c07c48e6badd5d44262916b2e19bed4751d1271c0f2c2727c2fe72aeaf0"}
Jul 08 16:34:37.395079 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.395062    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:37.395729 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.395712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bbnng"
Jul 08 16:34:37.428595 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:37.428530    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq" podStartSLOduration=9.295951288 podStartE2EDuration="28.428513237s" totalImagesPullingTime="19.132561949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:37.42629455 +0000 UTC m=+27.644491059" watchObservedRunningTime="2026-07-08 16:34:37.428513237 +0000 UTC m=+27.646709746"
Jul 08 16:34:38.239397 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.239359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:38.240272 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:38.239508    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:38.397264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.397189    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:34:38.397420 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.397335    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:38.397420 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.397408    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:38.413665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.413624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:38.413815 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:38.413749    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:39.239313 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:39.239282    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:39.239475 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:39.239323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:39.239475 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:39.239408    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:39.239807 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:39.239529    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:39.398913 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:39.398881    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:34:40.043144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.042894    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v4744"]
Jul 08 16:34:40.043362 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.043252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:40.043405 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:40.043364    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:40.043896 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.043624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nb49f"]
Jul 08 16:34:40.043896 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.043746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:40.043896 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:40.043857    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:40.045311 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.045288    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zsdxr"]
Jul 08 16:34:40.045421 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.045361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:40.045465 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:40.045424    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:40.400956 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.400875    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 16:34:40.494107 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:40.494075    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:34:42.238641 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:42.238519    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:42.238641 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:42.238595    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:42.238641 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:42.238610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:42.239157 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:42.238705    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:42.239157 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:42.238801    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:42.239157 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:42.238878    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:42.406861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:42.406829    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="a2fe46c78d2d0ed77dc7b262e49ebba0a369aedc5940e030434243a44691988e" exitCode=0
Jul 08 16:34:42.407019 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:42.406893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"a2fe46c78d2d0ed77dc7b262e49ebba0a369aedc5940e030434243a44691988e"}
Jul 08 16:34:43.950249 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:43.950211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:43.950750 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:43.950260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:43.950750 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:43.950361    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:43.950750 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:43.950367    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:43.950750 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:43.950410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret podName:669d7b5f-cd22-4fdb-8c5a-8f11cda0533b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:15.950396365 +0000 UTC m=+66.168592851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret") pod "global-pull-secret-syncer-v4744" (UID: "669d7b5f-cd22-4fdb-8c5a-8f11cda0533b") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:43.950750 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:43.950428    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:15.950417994 +0000 UTC m=+66.168614481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:44.051253 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.051222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:44.051420 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.051354    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:44.051420 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.051366    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:44.051420 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.051375    2559 projected.go:196] Error preparing data for projected volume kube-api-access-k6lqh for pod openshift-network-diagnostics/network-check-target-zsdxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:44.051420 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.051419    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh podName:1e59ba59-b01e-49a4-a3cd-37c3d89a41ea nodeName:}" failed. No retries permitted until 2026-07-08 16:35:16.051406897 +0000 UTC m=+66.269603384 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k6lqh" (UniqueName: "kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh") pod "network-check-target-zsdxr" (UID: "1e59ba59-b01e-49a4-a3cd-37c3d89a41ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:44.239093 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.238986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:44.239093 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.239044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:44.239093 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.239058    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:44.239317 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.239124    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zsdxr" podUID="1e59ba59-b01e-49a4-a3cd-37c3d89a41ea"
Jul 08 16:34:44.239317 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.239173    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4744" podUID="669d7b5f-cd22-4fdb-8c5a-8f11cda0533b"
Jul 08 16:34:44.239317 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:44.239266    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nb49f" podUID="26860848-e85c-4136-9004-bb2c83bfe483"
Jul 08 16:34:44.413038 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.413000    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="fe0921a74889cf7e3ae8fda5aba76f97491c15cd625d1f71ec256b5fc87e18e4" exitCode=0
Jul 08 16:34:44.413183 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:44.413048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"fe0921a74889cf7e3ae8fda5aba76f97491c15cd625d1f71ec256b5fc87e18e4"}
Jul 08 16:34:45.163886 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.163815    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeReady"
Jul 08 16:34:45.164278 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.163963    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:34:45.204444 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.204413    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:34:45.208591 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.208548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.211401 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.211376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:34:45.211552 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.211434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:34:45.211552 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.211381    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r7vmb\""
Jul 08 16:34:45.211719 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.211704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:34:45.217644 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.217613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:34:45.219852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.219825    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gglb5"]
Jul 08 16:34:45.222819 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.222799    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:34:45.222933 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.222899    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.225477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.225448    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:34:45.225645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.225462    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qx96f\""
Jul 08 16:34:45.225645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.225462    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:34:45.226394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.226370    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9sr8f"]
Jul 08 16:34:45.232370 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.229823    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.233064 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.233044    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:34:45.233777 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.233746    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:34:45.233952 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.233936    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:34:45.234074 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.234019    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tghtj\""
Jul 08 16:34:45.234329 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.234299    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gglb5"]
Jul 08 16:34:45.239799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.239771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sr8f"]
Jul 08 16:34:45.361526 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.361683 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebea962f-1221-4e2c-8479-5bd75507fe4b-tmp-dir\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.361683 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361683 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361683 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmvd\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361726    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42448\" (UniqueName: \"kubernetes.io/projected/ebea962f-1221-4e2c-8479-5bd75507fe4b-kube-api-access-42448\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqls\" (UniqueName: \"kubernetes.io/projected/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-kube-api-access-ndqls\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-cert\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361846    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.361940 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.362168 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.361962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebea962f-1221-4e2c-8479-5bd75507fe4b-config-volume\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.417829 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.417744    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="4c87033883736d5566dbea631baf3e3a7a28b1b572ecfc6faa6e515ea6120e98" exitCode=0
Jul 08 16:34:45.417941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.417831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"4c87033883736d5566dbea631baf3e3a7a28b1b572ecfc6faa6e515ea6120e98"}
Jul 08 16:34:45.432838 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.432797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bbnng_5ec01533-0ac2-429d-aa0d-b016eed8f657/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:45.463102 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463102 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463100    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmvd\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42448\" (UniqueName: \"kubernetes.io/projected/ebea962f-1221-4e2c-8479-5bd75507fe4b-kube-api-access-42448\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.463352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqls\" (UniqueName: \"kubernetes.io/projected/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-kube-api-access-ndqls\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.463352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-cert\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebea962f-1221-4e2c-8479-5bd75507fe4b-config-volume\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463565    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebea962f-1221-4e2c-8479-5bd75507fe4b-tmp-dir\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.463686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.464012 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.463684    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.464012 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.463762    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:45.464012 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.463777    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:34:45.464012 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.463841    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:45.96382146 +0000 UTC m=+36.182017964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:34:45.464179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.464089    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.464179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.464095    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebea962f-1221-4e2c-8479-5bd75507fe4b-tmp-dir\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.464179 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.463767    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:45.464179 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.464161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:45.96414583 +0000 UTC m=+36.182342339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:45.464344 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.464258    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ebea962f-1221-4e2c-8479-5bd75507fe4b-config-volume\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.464483 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.464453    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.467406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.467382    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-cert\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.467406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.467397    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.467554 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.467397    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.474183 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.474151    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqls\" (UniqueName: \"kubernetes.io/projected/3d7d00a0-116b-44f1-94b0-1d5a053a99a0-kube-api-access-ndqls\") pod \"ingress-canary-9sr8f\" (UID: \"3d7d00a0-116b-44f1-94b0-1d5a053a99a0\") " pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.475048 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.475022    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42448\" (UniqueName: \"kubernetes.io/projected/ebea962f-1221-4e2c-8479-5bd75507fe4b-kube-api-access-42448\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.475642 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.475624    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmvd\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.476079 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.476063    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.543152 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.543108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9sr8f"
Jul 08 16:34:45.593423 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.593377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bbnng_5ec01533-0ac2-429d-aa0d-b016eed8f657/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:45.685024 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.684948    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9sr8f"]
Jul 08 16:34:45.688930 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:45.688898    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7d00a0_116b_44f1_94b0_1d5a053a99a0.slice/crio-9e510adac82701c980096b9c3fe81905c82a3344594c67b1b13abe7e53511a89 WatchSource:0}: Error finding container 9e510adac82701c980096b9c3fe81905c82a3344594c67b1b13abe7e53511a89: Status 404 returned error can't find the container with id 9e510adac82701c980096b9c3fe81905c82a3344594c67b1b13abe7e53511a89
Jul 08 16:34:45.968165 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.968058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:45.968165 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:45.968103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:45.968376 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.968226    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:45.968376 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.968250    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:45.968376 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.968265    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:34:45.968376 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.968295    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:46.968279029 +0000 UTC m=+37.186475516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:45.968376 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:45.968318    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:46.968302132 +0000 UTC m=+37.186498635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:34:46.238833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.238738    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:34:46.239714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.238738    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:34:46.239714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.238738    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:34:46.242699 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.242679    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:34:46.242837 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.242822    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:34:46.243412 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.243396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:34:46.244064 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.244048    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g78k4\""
Jul 08 16:34:46.246454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.246256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zs6bx\""
Jul 08 16:34:46.246782 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.246749    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:34:46.420915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.420878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sr8f" event={"ID":"3d7d00a0-116b-44f1-94b0-1d5a053a99a0","Type":"ContainerStarted","Data":"9e510adac82701c980096b9c3fe81905c82a3344594c67b1b13abe7e53511a89"}
Jul 08 16:34:46.976865 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.976671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:46.977072 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:46.976886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:46.977072 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:46.976800    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:46.977072 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:46.977003    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:48.976987573 +0000 UTC m=+39.195184060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:46.977072 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:46.977050    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:46.977072 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:46.977065    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:34:46.977289 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:46.977119    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:48.977104452 +0000 UTC m=+39.195300955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:34:48.994852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:48.994813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:48.994870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:48.994983    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:48.995034    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:48.995048    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:48.995063    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.995040443 +0000 UTC m=+43.213236932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:48.995350 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:48.995083    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:52.995072834 +0000 UTC m=+43.213269343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:34:49.432135 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:49.432100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9sr8f" event={"ID":"3d7d00a0-116b-44f1-94b0-1d5a053a99a0","Type":"ContainerStarted","Data":"51b4d2fb6783a5bdce118776d7ab1960f23d2b3b0f8229dc72fdce540006d9ef"}
Jul 08 16:34:49.454059 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:49.454007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9sr8f" podStartSLOduration=1.048766918 podStartE2EDuration="4.45398915s" totalImagesPullingTime="3.405222232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:49.452905542 +0000 UTC m=+39.671102051" watchObservedRunningTime="2026-07-08 16:34:49.45398915 +0000 UTC m=+39.672185694"
Jul 08 16:34:53.026714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:53.026673    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:34:53.026714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:53.026723    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:34:53.027182 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:53.026857    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:53.027182 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:53.026861    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:53.027182 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:53.026933    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:01.026916016 +0000 UTC m=+51.245112503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:53.027182 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:53.026868    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:34:53.027182 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:53.027018    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:01.027006737 +0000 UTC m=+51.245203227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:34:53.445781 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:53.445740    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="0019ee09dd4b60054a89d7deba00128b2011eca7c9b98265ecfa7da9563dd193" exitCode=0
Jul 08 16:34:53.445955 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:53.445804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"0019ee09dd4b60054a89d7deba00128b2011eca7c9b98265ecfa7da9563dd193"}
Jul 08 16:34:54.451193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:54.451155    2559 generic.go:356] "Generic (PLEG): container finished" podID="81401226-bc48-426d-878e-bca23eb723e4" containerID="85fd436063d7aa65989020e1c62af78283d80603c8698b1e3010e4c277683a1a" exitCode=0
Jul 08 16:34:54.451591 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:54.451211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerDied","Data":"85fd436063d7aa65989020e1c62af78283d80603c8698b1e3010e4c277683a1a"}
Jul 08 16:34:55.456857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:55.456816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stzpv" event={"ID":"81401226-bc48-426d-878e-bca23eb723e4","Type":"ContainerStarted","Data":"301020ea879c64bca219be6113080e0ac74da27e622d3c5740c17018db95dc2f"}
Jul 08 16:34:55.504245 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:55.504187    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-stzpv" podStartSLOduration=9.661082786 podStartE2EDuration="46.504169363s" totalImagesPullingTime="36.843086577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:55.493946965 +0000 UTC m=+45.712143473" watchObservedRunningTime="2026-07-08 16:34:55.504169363 +0000 UTC m=+45.722365873"
Jul 08 16:34:56.649505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.649468    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"]
Jul 08 16:34:56.668108 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.668077    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.672088 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.672062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 08 16:34:56.672251 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.672223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.672411 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.672063    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.673009 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.672993    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fprtl\""
Jul 08 16:34:56.673105 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.673074    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 08 16:34:56.673159 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.673142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"]
Jul 08 16:34:56.753747 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.753712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62ddd43e-d605-4675-83e9-55704803d715-config\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.753937 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.753843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62ddd43e-d605-4675-83e9-55704803d715-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.753937 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.753863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp8n\" (UniqueName: \"kubernetes.io/projected/62ddd43e-d605-4675-83e9-55704803d715-kube-api-access-nnp8n\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.777916 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.777880    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"]
Jul 08 16:34:56.799795 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.799716    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"]
Jul 08 16:34:56.799956 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.799803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.813280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.813254    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.813431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.813344    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 08 16:34:56.813431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.813344    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 08 16:34:56.813431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.813411    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v77fd\""
Jul 08 16:34:56.813703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.813689    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.819807 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.819778    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"]
Jul 08 16:34:56.819950 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.819925    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.823832 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.823811    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.824021 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.824002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 08 16:34:56.824286 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.824265    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 08 16:34:56.824780 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.824761    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.826900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.826703    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-n4bsb\""
Jul 08 16:34:56.842088 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.842057    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"]
Jul 08 16:34:56.842088 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.842090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"]
Jul 08 16:34:56.842298 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.842105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"]
Jul 08 16:34:56.842298 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.842217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"
Jul 08 16:34:56.846069 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.846048    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.846069 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.846061    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.846640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.846624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zvcll\""
Jul 08 16:34:56.851267 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.851247    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"]
Jul 08 16:34:56.854132 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610465db-6b6f-4151-ba7a-08a61e6ede7c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.854235 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/610465db-6b6f-4151-ba7a-08a61e6ede7c-kube-api-access-448qr\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.854309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62ddd43e-d605-4675-83e9-55704803d715-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.854309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp8n\" (UniqueName: \"kubernetes.io/projected/62ddd43e-d605-4675-83e9-55704803d715-kube-api-access-nnp8n\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.854417 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5xxl\" (UniqueName: \"kubernetes.io/projected/bf60097d-f13a-4efe-a406-e0d9f389f29e-kube-api-access-j5xxl\") pod \"volume-data-source-validator-5d456fdbcd-2kj9p\" (UID: \"bf60097d-f13a-4efe-a406-e0d9f389f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"
Jul 08 16:34:56.854417 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62ddd43e-d605-4675-83e9-55704803d715-config\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.854417 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95193103-a1f6-4121-a136-78b16a76b0cb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.854417 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.854560 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/610465db-6b6f-4151-ba7a-08a61e6ede7c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.854560 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854470    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8b2\" (UniqueName: \"kubernetes.io/projected/95193103-a1f6-4121-a136-78b16a76b0cb-kube-api-access-pz8b2\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.854937 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.854918    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62ddd43e-d605-4675-83e9-55704803d715-config\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.857818 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.857798    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62ddd43e-d605-4675-83e9-55704803d715-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.868308 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.868282    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp8n\" (UniqueName: \"kubernetes.io/projected/62ddd43e-d605-4675-83e9-55704803d715-kube-api-access-nnp8n\") pod \"service-ca-operator-5d46d9ffd4-65gf9\" (UID: \"62ddd43e-d605-4675-83e9-55704803d715\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.875492 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.875469    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jcfjt"]
Jul 08 16:34:56.875643 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.875628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:56.878407 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.878376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 08 16:34:56.878715 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.878695    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c6txr\""
Jul 08 16:34:56.878857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.878720    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.878857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.878725    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.897193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.896178    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-vj8l6"]
Jul 08 16:34:56.897466 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.897452    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.909013 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.908554    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.909013 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.908721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pdj4h\""
Jul 08 16:34:56.909013 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.908863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 08 16:34:56.909013 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.908980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.911368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.911345    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 08 16:34:56.917229 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.917205    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 08 16:34:56.918352 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.918332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jcfjt"]
Jul 08 16:34:56.918431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.918374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"]
Jul 08 16:34:56.918431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.918398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-vj8l6"]
Jul 08 16:34:56.918513 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.918459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:56.925390 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.925367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 08 16:34:56.925885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.925870    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:56.925973 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.925865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 08 16:34:56.926804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.926790    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k9bc5\""
Jul 08 16:34:56.927080 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.927068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:56.942298 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.942276    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 08 16:34:56.955175 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/610465db-6b6f-4151-ba7a-08a61e6ede7c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.955324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8b2\" (UniqueName: \"kubernetes.io/projected/95193103-a1f6-4121-a136-78b16a76b0cb-kube-api-access-pz8b2\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.955324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62b9s\" (UniqueName: \"kubernetes.io/projected/7027dd97-8f1d-4c79-9429-8f9b6309bae3-kube-api-access-62b9s\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-trusted-ca\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610465db-6b6f-4151-ba7a-08a61e6ede7c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955356    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7027dd97-8f1d-4c79-9429-8f9b6309bae3-serving-cert\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-serving-cert\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955440    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-config\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-tmp\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:56.955493    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:56.955520 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5xxl\" (UniqueName: \"kubernetes.io/projected/bf60097d-f13a-4efe-a406-e0d9f389f29e-kube-api-access-j5xxl\") pod \"volume-data-source-validator-5d456fdbcd-2kj9p\" (UID: \"bf60097d-f13a-4efe-a406-e0d9f389f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdtw\" (UniqueName: \"kubernetes.io/projected/52100237-e2e8-4c8b-80af-247608c44c07-kube-api-access-vxdtw\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:56.955606    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.455562543 +0000 UTC m=+47.673759032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6mh5\" (UniqueName: \"kubernetes.io/projected/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-kube-api-access-v6mh5\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95193103-a1f6-4121-a136-78b16a76b0cb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-snapshots\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:56.955874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/610465db-6b6f-4151-ba7a-08a61e6ede7c-kube-api-access-448qr\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.956121 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:56.956121 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.955988    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/610465db-6b6f-4151-ba7a-08a61e6ede7c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.956358 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.956336    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/95193103-a1f6-4121-a136-78b16a76b0cb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.957884 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.957863    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/610465db-6b6f-4151-ba7a-08a61e6ede7c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.976589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.976546    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8b2\" (UniqueName: \"kubernetes.io/projected/95193103-a1f6-4121-a136-78b16a76b0cb-kube-api-access-pz8b2\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:56.978198 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.978178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"
Jul 08 16:34:56.980870 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.980838    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-448qr\" (UniqueName: \"kubernetes.io/projected/610465db-6b6f-4151-ba7a-08a61e6ede7c-kube-api-access-448qr\") pod \"kube-storage-version-migrator-operator-846bf668f7-6c6v5\" (UID: \"610465db-6b6f-4151-ba7a-08a61e6ede7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:56.982400 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:56.982380    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5xxl\" (UniqueName: \"kubernetes.io/projected/bf60097d-f13a-4efe-a406-e0d9f389f29e-kube-api-access-j5xxl\") pod \"volume-data-source-validator-5d456fdbcd-2kj9p\" (UID: \"bf60097d-f13a-4efe-a406-e0d9f389f29e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"
Jul 08 16:34:57.007088 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.007062    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5658d689b9-4zvzx"]
Jul 08 16:34:57.024914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.024887    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.029809 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.029774    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 16:34:57.030355 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.030332    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 08 16:34:57.030488 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.030412    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 16:34:57.031318 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.031298    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 08 16:34:57.031425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.031402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 08 16:34:57.031556 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.031541    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pfbhz\""
Jul 08 16:34:57.031669 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.031653    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 08 16:34:57.036268 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.036247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5658d689b9-4zvzx"]
Jul 08 16:34:57.057309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-trusted-ca\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.057469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.057469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7027dd97-8f1d-4c79-9429-8f9b6309bae3-serving-cert\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.057469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-serving-cert\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.057469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-config\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-tmp\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-default-certificate\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwn94\" (UniqueName: \"kubernetes.io/projected/7347fcff-b97d-48df-a5a1-b39adcae0f7b-kube-api-access-nwn94\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdtw\" (UniqueName: \"kubernetes.io/projected/52100237-e2e8-4c8b-80af-247608c44c07-kube-api-access-vxdtw\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:57.057708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-stats-auth\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6mh5\" (UniqueName: \"kubernetes.io/projected/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-kube-api-access-v6mh5\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-snapshots\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.058014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.057942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62b9s\" (UniqueName: \"kubernetes.io/projected/7027dd97-8f1d-4c79-9429-8f9b6309bae3-kube-api-access-62b9s\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.059000    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.059163    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-config\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.059425    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-trusted-ca\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.059523    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.059628    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls podName:52100237-e2e8-4c8b-80af-247608c44c07 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.55961001 +0000 UTC m=+47.777806512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6bprr" (UID: "52100237-e2e8-4c8b-80af-247608c44c07") : secret "samples-operator-tls" not found
Jul 08 16:34:57.060075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.059954    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-tmp\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.061020 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.060122    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7027dd97-8f1d-4c79-9429-8f9b6309bae3-snapshots\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.061020 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.060619    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7027dd97-8f1d-4c79-9429-8f9b6309bae3-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.061361 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.061340    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7027dd97-8f1d-4c79-9429-8f9b6309bae3-serving-cert\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.062396 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.062367    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-serving-cert\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.070483 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.069907    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62b9s\" (UniqueName: \"kubernetes.io/projected/7027dd97-8f1d-4c79-9429-8f9b6309bae3-kube-api-access-62b9s\") pod \"insights-operator-5c847dcfc4-jcfjt\" (UID: \"7027dd97-8f1d-4c79-9429-8f9b6309bae3\") " pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.072613 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.072159    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6mh5\" (UniqueName: \"kubernetes.io/projected/39e92342-e9d6-4611-94fa-cd53e1d5bbd9-kube-api-access-v6mh5\") pod \"console-operator-68b757fb67-vj8l6\" (UID: \"39e92342-e9d6-4611-94fa-cd53e1d5bbd9\") " pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.073239 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.073217    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdtw\" (UniqueName: \"kubernetes.io/projected/52100237-e2e8-4c8b-80af-247608c44c07-kube-api-access-vxdtw\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:57.129718 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.129684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"
Jul 08 16:34:57.147857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.147835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9"]
Jul 08 16:34:57.149234 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:57.149209    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ddd43e_d605_4675_83e9_55704803d715.slice/crio-6026e26e0a52cc59bf6f5707e84fb94244f5ba33f6b1b9f7177bf970350ddb83 WatchSource:0}: Error finding container 6026e26e0a52cc59bf6f5707e84fb94244f5ba33f6b1b9f7177bf970350ddb83: Status 404 returned error can't find the container with id 6026e26e0a52cc59bf6f5707e84fb94244f5ba33f6b1b9f7177bf970350ddb83
Jul 08 16:34:57.150060 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.150043    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"
Jul 08 16:34:57.159070 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.159003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-default-certificate\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.159070 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.159040    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwn94\" (UniqueName: \"kubernetes.io/projected/7347fcff-b97d-48df-a5a1-b39adcae0f7b-kube-api-access-nwn94\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.159070 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.159064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-stats-auth\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.159273 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.159253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.159348 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.159335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.159406 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.159394    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.65937763 +0000 UTC m=+47.877574118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:57.159464 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.159454    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:57.159525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.159514    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.659497599 +0000 UTC m=+47.877694101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:34:57.161625 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.161605    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-stats-auth\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.161720 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.161611    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-default-certificate\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.176385 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.176362    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwn94\" (UniqueName: \"kubernetes.io/projected/7347fcff-b97d-48df-a5a1-b39adcae0f7b-kube-api-access-nwn94\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.213350 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.213324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt"
Jul 08 16:34:57.227830 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.227746    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:34:57.324703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.324619    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5"]
Jul 08 16:34:57.332156 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:57.331717    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610465db_6b6f_4151_ba7a_08a61e6ede7c.slice/crio-6c4ca13569c376b31937dbd0945c2d9b523ba1e3c08e92c31376b1f5d572e573 WatchSource:0}: Error finding container 6c4ca13569c376b31937dbd0945c2d9b523ba1e3c08e92c31376b1f5d572e573: Status 404 returned error can't find the container with id 6c4ca13569c376b31937dbd0945c2d9b523ba1e3c08e92c31376b1f5d572e573
Jul 08 16:34:57.377407 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.377381    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p"]
Jul 08 16:34:57.380171 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:57.380137    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf60097d_f13a_4efe_a406_e0d9f389f29e.slice/crio-f7ffc5e4b8ef0b582a319b247dfb5914d0fe9297ca7f18a6c301afb56ef2b184 WatchSource:0}: Error finding container f7ffc5e4b8ef0b582a319b247dfb5914d0fe9297ca7f18a6c301afb56ef2b184: Status 404 returned error can't find the container with id f7ffc5e4b8ef0b582a319b247dfb5914d0fe9297ca7f18a6c301afb56ef2b184
Jul 08 16:34:57.397852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.397825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-jcfjt"]
Jul 08 16:34:57.401152 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:57.401124    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7027dd97_8f1d_4c79_9429_8f9b6309bae3.slice/crio-3147c1507dd9e4badc16cdb6adca137e145a605c258064bca645b324557f6050 WatchSource:0}: Error finding container 3147c1507dd9e4badc16cdb6adca137e145a605c258064bca645b324557f6050: Status 404 returned error can't find the container with id 3147c1507dd9e4badc16cdb6adca137e145a605c258064bca645b324557f6050
Jul 08 16:34:57.419438 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.419357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-vj8l6"]
Jul 08 16:34:57.421912 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:34:57.421888    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e92342_e9d6_4611_94fa_cd53e1d5bbd9.slice/crio-23f6425fcb40ec0d05ae7d2f478527143857ee17aa752d3ca0f4544c76dfff4f WatchSource:0}: Error finding container 23f6425fcb40ec0d05ae7d2f478527143857ee17aa752d3ca0f4544c76dfff4f: Status 404 returned error can't find the container with id 23f6425fcb40ec0d05ae7d2f478527143857ee17aa752d3ca0f4544c76dfff4f
Jul 08 16:34:57.461646 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.461613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:57.461873 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.461715    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:57.461873 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.461785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.461768009 +0000 UTC m=+48.679964497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:57.462598 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.462550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" event={"ID":"39e92342-e9d6-4611-94fa-cd53e1d5bbd9","Type":"ContainerStarted","Data":"23f6425fcb40ec0d05ae7d2f478527143857ee17aa752d3ca0f4544c76dfff4f"}
Jul 08 16:34:57.463556 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.463534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt" event={"ID":"7027dd97-8f1d-4c79-9429-8f9b6309bae3","Type":"ContainerStarted","Data":"3147c1507dd9e4badc16cdb6adca137e145a605c258064bca645b324557f6050"}
Jul 08 16:34:57.465175 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.465157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5" event={"ID":"610465db-6b6f-4151-ba7a-08a61e6ede7c","Type":"ContainerStarted","Data":"6c4ca13569c376b31937dbd0945c2d9b523ba1e3c08e92c31376b1f5d572e573"}
Jul 08 16:34:57.466195 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.466172    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9" event={"ID":"62ddd43e-d605-4675-83e9-55704803d715","Type":"ContainerStarted","Data":"6026e26e0a52cc59bf6f5707e84fb94244f5ba33f6b1b9f7177bf970350ddb83"}
Jul 08 16:34:57.467202 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.467182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p" event={"ID":"bf60097d-f13a-4efe-a406-e0d9f389f29e","Type":"ContainerStarted","Data":"f7ffc5e4b8ef0b582a319b247dfb5914d0fe9297ca7f18a6c301afb56ef2b184"}
Jul 08 16:34:57.562874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.562835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:57.563030 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.562970    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:57.563030 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.563023    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls podName:52100237-e2e8-4c8b-80af-247608c44c07 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.563008986 +0000 UTC m=+48.781205473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6bprr" (UID: "52100237-e2e8-4c8b-80af-247608c44c07") : secret "samples-operator-tls" not found
Jul 08 16:34:57.663895 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.663861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.664305 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:57.663915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:57.664305 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.664003    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:57.664305 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.664058    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.664030122 +0000 UTC m=+48.882226629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:57.664305 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:57.664091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:34:58.664079014 +0000 UTC m=+48.882275503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:34:58.472525 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:58.471979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:34:58.472525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.472158    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:58.472525 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.472219    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.472203386 +0000 UTC m=+50.690399872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:34:58.573381 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:58.573332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:34:58.573671 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.573654    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:34:58.573780 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.573725    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls podName:52100237-e2e8-4c8b-80af-247608c44c07 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.573704977 +0000 UTC m=+50.791901482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6bprr" (UID: "52100237-e2e8-4c8b-80af-247608c44c07") : secret "samples-operator-tls" not found
Jul 08 16:34:58.674589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:58.674546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:58.675044 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:34:58.674718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:34:58.675044 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.674879    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.674857748 +0000 UTC m=+50.893054257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:34:58.675479 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.675300    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:34:58.675479 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:34:58.675362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.675344913 +0000 UTC m=+50.893541406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:35:00.493214 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:00.493180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:00.493633 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.493317    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:00.493633 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.493381    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.493363127 +0000 UTC m=+54.711559631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:00.594657 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:00.594619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:35:00.594834 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.594795    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:00.594895 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.594883    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls podName:52100237-e2e8-4c8b-80af-247608c44c07 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.594860438 +0000 UTC m=+54.813056948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6bprr" (UID: "52100237-e2e8-4c8b-80af-247608c44c07") : secret "samples-operator-tls" not found
Jul 08 16:35:00.696006 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:00.695960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:00.696221 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:00.696047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:00.696221 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.696180    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.696160238 +0000 UTC m=+54.914356729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:00.696221 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.696212    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:35:00.696450 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:00.696276    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.696262834 +0000 UTC m=+54.914459342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:35:01.099259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:01.099218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:01.099320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:01.099351    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:01.099402    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:01.099412    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6985974f47-dfhf7: secret "image-registry-tls" not found
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:01.099413    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls podName:ebea962f-1221-4e2c-8479-5bd75507fe4b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:17.099399001 +0000 UTC m=+67.317595488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls") pod "dns-default-gglb5" (UID: "ebea962f-1221-4e2c-8479-5bd75507fe4b") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:01.099482 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:01.099456    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls podName:7170501a-3805-4eb1-9475-9f838c24ba45 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:17.099445952 +0000 UTC m=+67.317642440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls") pod "image-registry-6985974f47-dfhf7" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45") : secret "image-registry-tls" not found
Jul 08 16:35:02.487783 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.487745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5" event={"ID":"610465db-6b6f-4151-ba7a-08a61e6ede7c","Type":"ContainerStarted","Data":"239f8306515fb24e1759afc82a5a172fd264c592103236a5627a2c80c0f72be7"}
Jul 08 16:35:02.490485 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.490455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9" event={"ID":"62ddd43e-d605-4675-83e9-55704803d715","Type":"ContainerStarted","Data":"313ed5abcf4b8c761b6bdbf49eddba758a7c6a0d58d9407fb81de3c9c0f84977"}
Jul 08 16:35:02.492543 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.492514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p" event={"ID":"bf60097d-f13a-4efe-a406-e0d9f389f29e","Type":"ContainerStarted","Data":"acc562ddbc273d8b7979eca265caae39c3b06ce6b530485cbf99cc58b377cea6"}
Jul 08 16:35:02.523934 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.523872    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5" podStartSLOduration=1.567405665 podStartE2EDuration="6.523851879s" totalImagesPullingTime="4.956446214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:02.523519143 +0000 UTC m=+52.741715666" watchObservedRunningTime="2026-07-08 16:35:02.523851879 +0000 UTC m=+52.742048390"
Jul 08 16:35:02.561403 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.561337    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9" podStartSLOduration=1.4217537 podStartE2EDuration="6.561311106s" totalImagesPullingTime="5.139557406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:02.559989549 +0000 UTC m=+52.778186059" watchObservedRunningTime="2026-07-08 16:35:02.561311106 +0000 UTC m=+52.779507624"
Jul 08 16:35:02.590564 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:02.590448    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2kj9p" podStartSLOduration=1.683254918 podStartE2EDuration="6.590422239s" totalImagesPullingTime="4.907167321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:02.589901037 +0000 UTC m=+52.808097545" watchObservedRunningTime="2026-07-08 16:35:02.590422239 +0000 UTC m=+52.808618749"
Jul 08 16:35:03.489811 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.489775    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t9d9h"]
Jul 08 16:35:03.497511 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.497486    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/0.log"
Jul 08 16:35:03.497705 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.497526    2559 generic.go:356] "Generic (PLEG): container finished" podID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9" containerID="df5825979ac1ad25096baab8f775a96317ab657598c800a87d41149bbcc37773" exitCode=255
Jul 08 16:35:03.498213 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.498190    2559 scope.go:117] "RemoveContainer" containerID="df5825979ac1ad25096baab8f775a96317ab657598c800a87d41149bbcc37773"
Jul 08 16:35:03.514534 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.514508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9d9h"]
Jul 08 16:35:03.514666 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.514561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" event={"ID":"39e92342-e9d6-4611-94fa-cd53e1d5bbd9","Type":"ContainerDied","Data":"df5825979ac1ad25096baab8f775a96317ab657598c800a87d41149bbcc37773"}
Jul 08 16:35:03.514666 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.514612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt" event={"ID":"7027dd97-8f1d-4c79-9429-8f9b6309bae3","Type":"ContainerStarted","Data":"b9d7e1c8638e31002b3f4446421847cf94d5e79f06d52d1ccb82a10c12102432"}
Jul 08 16:35:03.514666 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.514630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.518450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.518266    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:35:03.518450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.518294    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:35:03.518450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.518311    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cshmv\""
Jul 08 16:35:03.575503 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.575448    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt" podStartSLOduration=2.683443703 podStartE2EDuration="7.575426533s" totalImagesPullingTime="4.89198283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:03.573910411 +0000 UTC m=+53.792106944" watchObservedRunningTime="2026-07-08 16:35:03.575426533 +0000 UTC m=+53.793623076"
Jul 08 16:35:03.629480 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.629370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.629904 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.629505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.629904 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.629542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lg28\" (UniqueName: \"kubernetes.io/projected/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-api-access-4lg28\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.629904 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.629617    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa362c9-df89-4ef1-86f3-464642a161f4-data-volume\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.629904 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.629680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffa362c9-df89-4ef1-86f3-464642a161f4-crio-socket\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.730956 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.730909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffa362c9-df89-4ef1-86f3-464642a161f4-crio-socket\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731124 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731124 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731206 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731124    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffa362c9-df89-4ef1-86f3-464642a161f4-crio-socket\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731206 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lg28\" (UniqueName: \"kubernetes.io/projected/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-api-access-4lg28\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731300 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:03.731210    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.731300 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa362c9-df89-4ef1-86f3-464642a161f4-data-volume\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.731300 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:03.731253    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:03.731300 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:03.731273    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.2312534 +0000 UTC m=+54.449449888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.731454 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:03.731317    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.231298198 +0000 UTC m=+54.449494696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:03.731640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.731618    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffa362c9-df89-4ef1-86f3-464642a161f4-data-volume\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:03.740567 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:03.740540    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lg28\" (UniqueName: \"kubernetes.io/projected/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-api-access-4lg28\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:04.236989 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.236951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:04.237207 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.237038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:04.237207 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.237119    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.237207 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.237171    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.237327 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.237210    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:05.237192 +0000 UTC m=+55.455388488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.237327 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.237238    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:05.23721837 +0000 UTC m=+55.455414872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.504697 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.504620    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:35:04.505151 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.505068    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/0.log"
Jul 08 16:35:04.505151 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.505113    2559 generic.go:356] "Generic (PLEG): container finished" podID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9" containerID="79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df" exitCode=255
Jul 08 16:35:04.505260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.505200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" event={"ID":"39e92342-e9d6-4611-94fa-cd53e1d5bbd9","Type":"ContainerDied","Data":"79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df"}
Jul 08 16:35:04.505316 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.505263    2559 scope.go:117] "RemoveContainer" containerID="df5825979ac1ad25096baab8f775a96317ab657598c800a87d41149bbcc37773"
Jul 08 16:35:04.505630 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.505596    2559 scope.go:117] "RemoveContainer" containerID="79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df"
Jul 08 16:35:04.505801 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.505779    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-vj8l6_openshift-console-operator(39e92342-e9d6-4611-94fa-cd53e1d5bbd9)\"" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" podUID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9"
Jul 08 16:35:04.539739 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.539695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:04.539964 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.539799    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:04.539964 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.539864    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.539845501 +0000 UTC m=+62.758041988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:04.641152 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.641113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:35:04.641347 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.641287    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 16:35:04.641422 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.641390    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls podName:52100237-e2e8-4c8b-80af-247608c44c07 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.641365925 +0000 UTC m=+62.859562429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6bprr" (UID: "52100237-e2e8-4c8b-80af-247608c44c07") : secret "samples-operator-tls" not found
Jul 08 16:35:04.742597 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.742537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:04.742811 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:04.742625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:04.742811 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.742708    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.742690076 +0000 UTC m=+62.960886568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:04.742811 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.742792    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:35:04.742975 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:04.742854    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:12.742838349 +0000 UTC m=+62.961034837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:35:05.247075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:05.247032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:05.247239 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:05.247112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:05.247239 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:05.247214    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:05.247239 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:05.247217    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:05.247378 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:05.247267    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.24725418 +0000 UTC m=+57.465450668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:05.247378 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:05.247281    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.247274474 +0000 UTC m=+57.465470961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:05.258646 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:05.258621    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d558v_25359783-c9a1-4d17-b19d-cd028d55d434/dns-node-resolver/0.log"
Jul 08 16:35:05.509605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:05.509504    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:35:05.509982 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:05.509887    2559 scope.go:117] "RemoveContainer" containerID="79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df"
Jul 08 16:35:05.510057 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:05.510038    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-vj8l6_openshift-console-operator(39e92342-e9d6-4611-94fa-cd53e1d5bbd9)\"" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" podUID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9"
Jul 08 16:35:06.458249 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:06.458220    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mdplr_797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1/node-ca/0.log"
Jul 08 16:35:07.060241 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.060203    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9sr8f_3d7d00a0-116b-44f1-94b0-1d5a053a99a0/serve-healthcheck-canary/0.log"
Jul 08 16:35:07.228787 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.228750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:35:07.228964 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.228844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:35:07.229192 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.229180    2559 scope.go:117] "RemoveContainer" containerID="79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df"
Jul 08 16:35:07.229351 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:07.229333    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-vj8l6_openshift-console-operator(39e92342-e9d6-4611-94fa-cd53e1d5bbd9)\"" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" podUID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9"
Jul 08 16:35:07.264496 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.264456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:07.264678 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:07.264524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:07.264678 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:07.264627    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:07.264678 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:07.264631    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:07.264780 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:07.264684    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.264670849 +0000 UTC m=+61.482867341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:07.264780 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:07.264697    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.264691723 +0000 UTC m=+61.482888209 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:11.302112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:11.302072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:11.302555 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:11.302175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:11.302555 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:11.302238    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:11.302555 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:11.302283    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:11.302555 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:11.302335    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.302314082 +0000 UTC m=+69.520510623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:11.302555 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:11.302357    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.302346911 +0000 UTC m=+69.520543399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:11.419791 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:11.419757    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqpfq"
Jul 08 16:35:12.615125 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.615084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:12.615527 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:12.615230    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:12.615527 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:12.615326    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls podName:95193103-a1f6-4121-a136-78b16a76b0cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:28.615308031 +0000 UTC m=+78.833504518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-ktzts" (UID: "95193103-a1f6-4121-a136-78b16a76b0cb") : secret "cluster-monitoring-operator-tls" not found
Jul 08 16:35:12.715804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.715765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:35:12.718324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.718291    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/52100237-e2e8-4c8b-80af-247608c44c07-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6bprr\" (UID: \"52100237-e2e8-4c8b-80af-247608c44c07\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:35:12.798094 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.798062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c6txr\""
Jul 08 16:35:12.805656 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.805631    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"
Jul 08 16:35:12.817162 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.817137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:12.817288 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.817225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:12.817348 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:12.817326    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:28.81730478 +0000 UTC m=+79.035501291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : configmap references non-existent config key: service-ca.crt
Jul 08 16:35:12.817418 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:12.817395    2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 08 16:35:12.817471 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:12.817462    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs podName:7347fcff-b97d-48df-a5a1-b39adcae0f7b nodeName:}" failed. No retries permitted until 2026-07-08 16:35:28.817446009 +0000 UTC m=+79.035642511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs") pod "router-default-5658d689b9-4zvzx" (UID: "7347fcff-b97d-48df-a5a1-b39adcae0f7b") : secret "router-metrics-certs-default" not found
Jul 08 16:35:12.937594 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:12.937548    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr"]
Jul 08 16:35:13.538110 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:13.538071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr" event={"ID":"52100237-e2e8-4c8b-80af-247608c44c07","Type":"ContainerStarted","Data":"9e7f53001e13c341a54e5849db922fa67e0eeab401fe9308fde82af7fb361ea0"}
Jul 08 16:35:15.545915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:15.545829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr" event={"ID":"52100237-e2e8-4c8b-80af-247608c44c07","Type":"ContainerStarted","Data":"014ac336565b1f546baccdb4693a6a5868dc190e956b63d0561ca15e3de15a90"}
Jul 08 16:35:15.545915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:15.545891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr" event={"ID":"52100237-e2e8-4c8b-80af-247608c44c07","Type":"ContainerStarted","Data":"efc847ddf33a0eefdd4bd97972fa5b73f94f8e49dd417b467a6b16c93d4d6f38"}
Jul 08 16:35:15.582843 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:15.582786    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6bprr" podStartSLOduration=17.275978448 podStartE2EDuration="19.582770868s" totalImagesPullingTime="2.30679242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:15.581860202 +0000 UTC m=+65.800056711" watchObservedRunningTime="2026-07-08 16:35:15.582770868 +0000 UTC m=+65.800967377"
Jul 08 16:35:16.050205 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.050169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:35:16.050406 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.050231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:35:16.055116 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.055093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:16.055169 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.055100    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:16.060615 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:16.060592    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 16:35:16.060728 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:16.060662    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs podName:26860848-e85c-4136-9004-bb2c83bfe483 nodeName:}" failed. No retries permitted until 2026-07-08 16:36:20.06064138 +0000 UTC m=+130.278837885 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs") pod "network-metrics-daemon-nb49f" (UID: "26860848-e85c-4136-9004-bb2c83bfe483") : secret "metrics-daemon-secret" not found
Jul 08 16:35:16.063023 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.062992    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/669d7b5f-cd22-4fdb-8c5a-8f11cda0533b-original-pull-secret\") pod \"global-pull-secret-syncer-v4744\" (UID: \"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b\") " pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:35:16.151258 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.151222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:35:16.154900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.154882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:16.165415 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.165390    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:16.174710 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.174681    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lqh\" (UniqueName: \"kubernetes.io/projected/1e59ba59-b01e-49a4-a3cd-37c3d89a41ea-kube-api-access-k6lqh\") pod \"network-check-target-zsdxr\" (UID: \"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea\") " pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:35:16.249660 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.249629    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4744"
Jul 08 16:35:16.264943 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.264911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zs6bx\""
Jul 08 16:35:16.271900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.271873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:35:16.399807 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.399768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v4744"]
Jul 08 16:35:16.402566 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:16.402538    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669d7b5f_cd22_4fdb_8c5a_8f11cda0533b.slice/crio-053ec29daf5dd825db8eda83d32cc4af6e0213aee8407f3ade759095e4f77245 WatchSource:0}: Error finding container 053ec29daf5dd825db8eda83d32cc4af6e0213aee8407f3ade759095e4f77245: Status 404 returned error can't find the container with id 053ec29daf5dd825db8eda83d32cc4af6e0213aee8407f3ade759095e4f77245
Jul 08 16:35:16.420524 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.420497    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zsdxr"]
Jul 08 16:35:16.424277 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:16.424252    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e59ba59_b01e_49a4_a3cd_37c3d89a41ea.slice/crio-a2af9da3a5f4ec8f5f5d8fb2a682b936b556090eb5c9659921896b8838673bbd WatchSource:0}: Error finding container a2af9da3a5f4ec8f5f5d8fb2a682b936b556090eb5c9659921896b8838673bbd: Status 404 returned error can't find the container with id a2af9da3a5f4ec8f5f5d8fb2a682b936b556090eb5c9659921896b8838673bbd
Jul 08 16:35:16.550144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.550103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zsdxr" event={"ID":"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea","Type":"ContainerStarted","Data":"a2af9da3a5f4ec8f5f5d8fb2a682b936b556090eb5c9659921896b8838673bbd"}
Jul 08 16:35:16.551279 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:16.551259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v4744" event={"ID":"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b","Type":"ContainerStarted","Data":"053ec29daf5dd825db8eda83d32cc4af6e0213aee8407f3ade759095e4f77245"}
Jul 08 16:35:17.160813 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.160774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:17.161016 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.160869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:17.164158 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.164128    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"image-registry-6985974f47-dfhf7\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") " pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:17.165226 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.165193    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebea962f-1221-4e2c-8479-5bd75507fe4b-metrics-tls\") pod \"dns-default-gglb5\" (UID: \"ebea962f-1221-4e2c-8479-5bd75507fe4b\") " pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:17.323893 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.323856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r7vmb\""
Jul 08 16:35:17.331892 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.331624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:17.340172 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.340144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qx96f\""
Jul 08 16:35:17.348093 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.348049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:17.518227 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.518189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:35:17.522295 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:17.522256    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7170501a_3805_4eb1_9475_9f838c24ba45.slice/crio-38a6c97dd9c6de66dac294fba40f5b4834570d3ec2963a57614b7defdd5438a3 WatchSource:0}: Error finding container 38a6c97dd9c6de66dac294fba40f5b4834570d3ec2963a57614b7defdd5438a3: Status 404 returned error can't find the container with id 38a6c97dd9c6de66dac294fba40f5b4834570d3ec2963a57614b7defdd5438a3
Jul 08 16:35:17.534845 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.534766    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gglb5"]
Jul 08 16:35:17.539419 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:17.539387    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebea962f_1221_4e2c_8479_5bd75507fe4b.slice/crio-e186057dcc691972a5d66e0e3c1a93f617a70b705d9e8c459d052f374a6ad6c4 WatchSource:0}: Error finding container e186057dcc691972a5d66e0e3c1a93f617a70b705d9e8c459d052f374a6ad6c4: Status 404 returned error can't find the container with id e186057dcc691972a5d66e0e3c1a93f617a70b705d9e8c459d052f374a6ad6c4
Jul 08 16:35:17.556191 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.556149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gglb5" event={"ID":"ebea962f-1221-4e2c-8479-5bd75507fe4b","Type":"ContainerStarted","Data":"e186057dcc691972a5d66e0e3c1a93f617a70b705d9e8c459d052f374a6ad6c4"}
Jul 08 16:35:17.557822 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:17.557790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" event={"ID":"7170501a-3805-4eb1-9475-9f838c24ba45","Type":"ContainerStarted","Data":"38a6c97dd9c6de66dac294fba40f5b4834570d3ec2963a57614b7defdd5438a3"}
Jul 08 16:35:18.564351 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:18.564314    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" event={"ID":"7170501a-3805-4eb1-9475-9f838c24ba45","Type":"ContainerStarted","Data":"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"}
Jul 08 16:35:18.565145 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:18.565120    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:18.592563 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:18.592491    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" podStartSLOduration=68.592468448 podStartE2EDuration="1m8.592468448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:18.58983077 +0000 UTC m=+68.808027292" watchObservedRunningTime="2026-07-08 16:35:18.592468448 +0000 UTC m=+68.810664938"
Jul 08 16:35:19.384339 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:19.384291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:19.384533 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:19.384408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:19.384533 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:19.384476    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.384694 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:19.384563    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls podName:ffa362c9-df89-4ef1-86f3-464642a161f4 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:35.384541416 +0000 UTC m=+85.602737926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9d9h" (UID: "ffa362c9-df89-4ef1-86f3-464642a161f4") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.384993 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:19.384964    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffa362c9-df89-4ef1-86f3-464642a161f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:21.238946 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.238911    2559 scope.go:117] "RemoveContainer" containerID="79562efea4b4204e250fb1da257cdb83a208f4d9a92e0f475593bee5896576df"
Jul 08 16:35:21.592037 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.592006    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:35:21.592195 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.592070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" event={"ID":"39e92342-e9d6-4611-94fa-cd53e1d5bbd9","Type":"ContainerStarted","Data":"2529c93ebea480af8f0ab470d27d52a2296088d2ca400650436cb1dec0a4364f"}
Jul 08 16:35:21.593543 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.593209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:35:21.594623 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.594567    2559 patch_prober.go:28] interesting pod/console-operator-68b757fb67-vj8l6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Jul 08 16:35:21.594744 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:21.594653    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" podUID="39e92342-e9d6-4611-94fa-cd53e1d5bbd9" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused"
Jul 08 16:35:22.597894 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.597854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gglb5" event={"ID":"ebea962f-1221-4e2c-8479-5bd75507fe4b","Type":"ContainerStarted","Data":"49fb5e70f24cb82064a8b2f6b4372e8507c85fb89fb4d793f0ddc80c0920b818"}
Jul 08 16:35:22.598327 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.597911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gglb5" event={"ID":"ebea962f-1221-4e2c-8479-5bd75507fe4b","Type":"ContainerStarted","Data":"26f33a67f20ed2828e3c793cb98939d8782fccafd4f9ec07af816bc7c54dadb2"}
Jul 08 16:35:22.598327 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.597942    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:22.599375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.599338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zsdxr" event={"ID":"1e59ba59-b01e-49a4-a3cd-37c3d89a41ea","Type":"ContainerStarted","Data":"a8da57eafc695911359da8e22ba17b27ce06b3028c73a8e4edd964808f844dff"}
Jul 08 16:35:22.599504 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.599404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:35:22.600782 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.600758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v4744" event={"ID":"669d7b5f-cd22-4fdb-8c5a-8f11cda0533b","Type":"ContainerStarted","Data":"f0cbb6199726b6438b31c3cde7b672951c5839cf527529dbdbc86c12691872be"}
Jul 08 16:35:22.605126 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.605103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6"
Jul 08 16:35:22.625988 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.625941    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-vj8l6" podStartSLOduration=21.759198432 podStartE2EDuration="26.625924554s" totalImagesPullingTime="4.866726122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:21.617379562 +0000 UTC m=+71.835576072" watchObservedRunningTime="2026-07-08 16:35:22.625924554 +0000 UTC m=+72.844121062"
Jul 08 16:35:22.626431 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.626407    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gglb5" podStartSLOduration=33.677899405 podStartE2EDuration="37.626398739s" totalImagesPullingTime="3.948499334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:22.625853101 +0000 UTC m=+72.844049609" watchObservedRunningTime="2026-07-08 16:35:22.626398739 +0000 UTC m=+72.844595225"
Jul 08 16:35:22.678608 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.678542    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v4744" podStartSLOduration=67.578174473 podStartE2EDuration="1m12.678524013s" totalImagesPullingTime="5.10034954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:22.677406213 +0000 UTC m=+72.895602713" watchObservedRunningTime="2026-07-08 16:35:22.678524013 +0000 UTC m=+72.896720521"
Jul 08 16:35:22.702298 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:22.702233    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zsdxr" podStartSLOduration=67.639449978 podStartE2EDuration="1m12.70221059s" totalImagesPullingTime="5.062760612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:22.701850904 +0000 UTC m=+72.920047415" watchObservedRunningTime="2026-07-08 16:35:22.70221059 +0000 UTC m=+72.920407096"
Jul 08 16:35:28.666800 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.666750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:28.669325 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.669300    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/95193103-a1f6-4121-a136-78b16a76b0cb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-ktzts\" (UID: \"95193103-a1f6-4121-a136-78b16a76b0cb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:28.868924 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.868874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:28.869115 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.868962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:28.869489 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.869453    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7347fcff-b97d-48df-a5a1-b39adcae0f7b-service-ca-bundle\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:28.871438 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.871407    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7347fcff-b97d-48df-a5a1-b39adcae0f7b-metrics-certs\") pod \"router-default-5658d689b9-4zvzx\" (UID: \"7347fcff-b97d-48df-a5a1-b39adcae0f7b\") " pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:28.911995 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.911955    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v77fd\""
Jul 08 16:35:28.919040 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:28.918969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"
Jul 08 16:35:29.052019 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.051984    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts"]
Jul 08 16:35:29.055206 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:29.055172    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95193103_a1f6_4121_a136_78b16a76b0cb.slice/crio-3c9ed3b6e26c137071799eeed494b41863ca3ede2257e0d8c5c9a86b77d60097 WatchSource:0}: Error finding container 3c9ed3b6e26c137071799eeed494b41863ca3ede2257e0d8c5c9a86b77d60097: Status 404 returned error can't find the container with id 3c9ed3b6e26c137071799eeed494b41863ca3ede2257e0d8c5c9a86b77d60097
Jul 08 16:35:29.138596 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.138542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-pfbhz\""
Jul 08 16:35:29.146248 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.146226    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:29.286790 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.286758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5658d689b9-4zvzx"]
Jul 08 16:35:29.290546 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:29.290517    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7347fcff_b97d_48df_a5a1_b39adcae0f7b.slice/crio-73b7b4f6538c1164aa66cb991b33f20680d6a43b40b725755ea8506b7fcfc9dd WatchSource:0}: Error finding container 73b7b4f6538c1164aa66cb991b33f20680d6a43b40b725755ea8506b7fcfc9dd: Status 404 returned error can't find the container with id 73b7b4f6538c1164aa66cb991b33f20680d6a43b40b725755ea8506b7fcfc9dd
Jul 08 16:35:29.629394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.629352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts" event={"ID":"95193103-a1f6-4121-a136-78b16a76b0cb","Type":"ContainerStarted","Data":"3c9ed3b6e26c137071799eeed494b41863ca3ede2257e0d8c5c9a86b77d60097"}
Jul 08 16:35:29.630824 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.630798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5658d689b9-4zvzx" event={"ID":"7347fcff-b97d-48df-a5a1-b39adcae0f7b","Type":"ContainerStarted","Data":"e29510550bd41031f8e83aee2ebf58b99d7124c3f0cf675dc13e3ecba72d916f"}
Jul 08 16:35:29.630937 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.630839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5658d689b9-4zvzx" event={"ID":"7347fcff-b97d-48df-a5a1-b39adcae0f7b","Type":"ContainerStarted","Data":"73b7b4f6538c1164aa66cb991b33f20680d6a43b40b725755ea8506b7fcfc9dd"}
Jul 08 16:35:29.657389 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:29.657334    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5658d689b9-4zvzx" podStartSLOduration=33.657315974 podStartE2EDuration="33.657315974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:29.657222279 +0000 UTC m=+79.875418787" watchObservedRunningTime="2026-07-08 16:35:29.657315974 +0000 UTC m=+79.875512482"
Jul 08 16:35:30.146615 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.146560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:30.149671 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.149644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:30.635649 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.635617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:30.637227 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.637193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5658d689b9-4zvzx"
Jul 08 16:35:30.656652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.656623    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-v54bj"]
Jul 08 16:35:30.659983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.659952    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:30.664454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.662254    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"]
Jul 08 16:35:30.664454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.663523    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:35:30.664454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.663966    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:35:30.664454 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.664177    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lrqft\""
Jul 08 16:35:30.666291 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.666269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.667002 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.666978    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-v54bj"]
Jul 08 16:35:30.670928 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.670721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mjk8r\""
Jul 08 16:35:30.670928 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.670818    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 08 16:35:30.674762 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.674742    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 08 16:35:30.681521 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.681496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgqz\" (UniqueName: \"kubernetes.io/projected/dd42600f-0b4c-49de-9456-215bcf497926-kube-api-access-lsgqz\") pod \"downloads-685bfb5f64-v54bj\" (UID: \"dd42600f-0b4c-49de-9456-215bcf497926\") " pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:30.681684 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.681600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ef29d45-9438-4c96-9e06-327eaf5fcacc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.681684 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.681668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ef29d45-9438-4c96-9e06-327eaf5fcacc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.682113 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.682090    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"]
Jul 08 16:35:30.782505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.782464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ef29d45-9438-4c96-9e06-327eaf5fcacc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.782723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.782555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgqz\" (UniqueName: \"kubernetes.io/projected/dd42600f-0b4c-49de-9456-215bcf497926-kube-api-access-lsgqz\") pod \"downloads-685bfb5f64-v54bj\" (UID: \"dd42600f-0b4c-49de-9456-215bcf497926\") " pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:30.782723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.782637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ef29d45-9438-4c96-9e06-327eaf5fcacc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.783353 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.783253    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ef29d45-9438-4c96-9e06-327eaf5fcacc-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.785427 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.785401    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ef29d45-9438-4c96-9e06-327eaf5fcacc-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-mt2zz\" (UID: \"6ef29d45-9438-4c96-9e06-327eaf5fcacc\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:30.794214 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.794185    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgqz\" (UniqueName: \"kubernetes.io/projected/dd42600f-0b4c-49de-9456-215bcf497926-kube-api-access-lsgqz\") pod \"downloads-685bfb5f64-v54bj\" (UID: \"dd42600f-0b4c-49de-9456-215bcf497926\") " pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:30.976717 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.976626    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:30.983365 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:30.983325    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"
Jul 08 16:35:31.321044 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.321007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz"]
Jul 08 16:35:31.324030 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:31.324005    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef29d45_9438_4c96_9e06_327eaf5fcacc.slice/crio-ece842efb0ab97bea53906b548ac49eff4986ffe21250da1c382fed4f739fe55 WatchSource:0}: Error finding container ece842efb0ab97bea53906b548ac49eff4986ffe21250da1c382fed4f739fe55: Status 404 returned error can't find the container with id ece842efb0ab97bea53906b548ac49eff4986ffe21250da1c382fed4f739fe55
Jul 08 16:35:31.346808 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.346763    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-v54bj"]
Jul 08 16:35:31.351227 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:31.351193    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd42600f_0b4c_49de_9456_215bcf497926.slice/crio-967143c940ab2f39b791220f0ca679abec35671801408099df89a5888da2ae5a WatchSource:0}: Error finding container 967143c940ab2f39b791220f0ca679abec35671801408099df89a5888da2ae5a: Status 404 returned error can't find the container with id 967143c940ab2f39b791220f0ca679abec35671801408099df89a5888da2ae5a
Jul 08 16:35:31.640240 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.640206    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-v54bj" event={"ID":"dd42600f-0b4c-49de-9456-215bcf497926","Type":"ContainerStarted","Data":"967143c940ab2f39b791220f0ca679abec35671801408099df89a5888da2ae5a"}
Jul 08 16:35:31.641710 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.641671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts" event={"ID":"95193103-a1f6-4121-a136-78b16a76b0cb","Type":"ContainerStarted","Data":"365d597d631958dd55e3322884ee22a14ee7e9ca6bd7ccfc891a9eac99eefd11"}
Jul 08 16:35:31.642840 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.642809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz" event={"ID":"6ef29d45-9438-4c96-9e06-327eaf5fcacc","Type":"ContainerStarted","Data":"ece842efb0ab97bea53906b548ac49eff4986ffe21250da1c382fed4f739fe55"}
Jul 08 16:35:31.660622 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.660535    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-ktzts" podStartSLOduration=33.483783791 podStartE2EDuration="35.660512638s" totalImagesPullingTime="2.176728847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:31.659631347 +0000 UTC m=+81.877827872" watchObservedRunningTime="2026-07-08 16:35:31.660512638 +0000 UTC m=+81.878709147"
Jul 08 16:35:31.825689 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.825652    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"]
Jul 08 16:35:31.858640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.858604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"]
Jul 08 16:35:31.858832 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.858729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:31.861543 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.861521    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 08 16:35:31.861711 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.861676    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-htr4v\""
Jul 08 16:35:31.893851 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.893752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9nvfr\" (UID: \"340a75ef-1b40-4dd0-a2af-4c1c27e49f06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:31.995372 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:31.995324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9nvfr\" (UID: \"340a75ef-1b40-4dd0-a2af-4c1c27e49f06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:31.995554 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:31.995481    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 08 16:35:31.995554 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:31.995551    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates podName:340a75ef-1b40-4dd0-a2af-4c1c27e49f06 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:32.495534816 +0000 UTC m=+82.713731303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-9nvfr" (UID: "340a75ef-1b40-4dd0-a2af-4c1c27e49f06") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 08 16:35:32.499469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:32.499419    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9nvfr\" (UID: \"340a75ef-1b40-4dd0-a2af-4c1c27e49f06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:32.503047 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:32.503018    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/340a75ef-1b40-4dd0-a2af-4c1c27e49f06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9nvfr\" (UID: \"340a75ef-1b40-4dd0-a2af-4c1c27e49f06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:32.606590 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:32.606532    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gglb5"
Jul 08 16:35:32.768987 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:32.768907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:32.942222 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:32.942190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"]
Jul 08 16:35:32.996728 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:32.996685    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340a75ef_1b40_4dd0_a2af_4c1c27e49f06.slice/crio-22d339e310405300ac1d90fc3a9379f529edb26746f5239960f54ff617c4b40b WatchSource:0}: Error finding container 22d339e310405300ac1d90fc3a9379f529edb26746f5239960f54ff617c4b40b: Status 404 returned error can't find the container with id 22d339e310405300ac1d90fc3a9379f529edb26746f5239960f54ff617c4b40b
Jul 08 16:35:33.657412 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:33.657377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz" event={"ID":"6ef29d45-9438-4c96-9e06-327eaf5fcacc","Type":"ContainerStarted","Data":"2104fa0c4966d06045cbf603a7d736a25b8b75ce0356e27dade30b51f4372c51"}
Jul 08 16:35:33.658560 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:33.658540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr" event={"ID":"340a75ef-1b40-4dd0-a2af-4c1c27e49f06","Type":"ContainerStarted","Data":"22d339e310405300ac1d90fc3a9379f529edb26746f5239960f54ff617c4b40b"}
Jul 08 16:35:33.678086 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:33.678022    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-mt2zz" podStartSLOduration=1.977465795 podStartE2EDuration="3.678001204s" totalImagesPullingTime="1.700535409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:33.675478655 +0000 UTC m=+83.893675173" watchObservedRunningTime="2026-07-08 16:35:33.678001204 +0000 UTC m=+83.896197712"
Jul 08 16:35:35.425928 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.425889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:35.429162 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.429133    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffa362c9-df89-4ef1-86f3-464642a161f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9d9h\" (UID: \"ffa362c9-df89-4ef1-86f3-464642a161f4\") " pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:35.635364 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.635326    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cshmv\""
Jul 08 16:35:35.642973 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.642942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9d9h"
Jul 08 16:35:35.670957 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.670920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr" event={"ID":"340a75ef-1b40-4dd0-a2af-4c1c27e49f06","Type":"ContainerStarted","Data":"ce3fea1038167cf793ff4b42509446fb6897bad718edb5e0ba5ba888a95602b1"}
Jul 08 16:35:35.671147 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.671118    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:35.677055 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.676949    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr"
Jul 08 16:35:35.715809 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.715747    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9nvfr" podStartSLOduration=3.12938029 podStartE2EDuration="4.715720984s" totalImagesPullingTime="1.586340694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:35.692918451 +0000 UTC m=+85.911114961" watchObservedRunningTime="2026-07-08 16:35:35.715720984 +0000 UTC m=+85.933917496"
Jul 08 16:35:35.799281 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.799255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9d9h"]
Jul 08 16:35:35.801581 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:35.801548    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa362c9_df89_4ef1_86f3_464642a161f4.slice/crio-8d534fc3c912293906224ff7aad269fc042f99aec61f04765cb6ee4bae9f3e0a WatchSource:0}: Error finding container 8d534fc3c912293906224ff7aad269fc042f99aec61f04765cb6ee4bae9f3e0a: Status 404 returned error can't find the container with id 8d534fc3c912293906224ff7aad269fc042f99aec61f04765cb6ee4bae9f3e0a
Jul 08 16:35:35.899759 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.899720    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"]
Jul 08 16:35:35.903974 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.903950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:35.907052 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.907027    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mchc5\""
Jul 08 16:35:35.907540 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.907524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 16:35:35.908656 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.908630    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 16:35:35.908656 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.908653    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:35:35.916070 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.916046    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"]
Jul 08 16:35:35.930772 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.930689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8vvb\" (UniqueName: \"kubernetes.io/projected/d59253a2-e27c-4715-9306-6e7399ea8980-kube-api-access-n8vvb\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:35.930772 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.930730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59253a2-e27c-4715-9306-6e7399ea8980-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:35.930772 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.930763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:35.931023 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:35.930892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.031912 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.031869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.032083 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.031931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8vvb\" (UniqueName: \"kubernetes.io/projected/d59253a2-e27c-4715-9306-6e7399ea8980-kube-api-access-n8vvb\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.032083 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.031955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59253a2-e27c-4715-9306-6e7399ea8980-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.032083 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.031974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.032227 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:36.032212    2559 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 08 16:35:36.032284 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:36.032275    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls podName:d59253a2-e27c-4715-9306-6e7399ea8980 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:36.532257552 +0000 UTC m=+86.750454061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-kjtvx" (UID: "d59253a2-e27c-4715-9306-6e7399ea8980") : secret "prometheus-operator-tls" not found
Jul 08 16:35:36.032837 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.032770    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59253a2-e27c-4715-9306-6e7399ea8980-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.034588 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.034549    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.041600 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.041561    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8vvb\" (UniqueName: \"kubernetes.io/projected/d59253a2-e27c-4715-9306-6e7399ea8980-kube-api-access-n8vvb\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.535696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.535662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.538408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.538382    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d59253a2-e27c-4715-9306-6e7399ea8980-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-kjtvx\" (UID: \"d59253a2-e27c-4715-9306-6e7399ea8980\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:36.676990 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.676935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9d9h" event={"ID":"ffa362c9-df89-4ef1-86f3-464642a161f4","Type":"ContainerStarted","Data":"8d534fc3c912293906224ff7aad269fc042f99aec61f04765cb6ee4bae9f3e0a"}
Jul 08 16:35:36.816823 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:36.816737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"
Jul 08 16:35:37.019997 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:37.019791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx"]
Jul 08 16:35:37.023605 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:37.023546    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59253a2_e27c_4715_9306_6e7399ea8980.slice/crio-5c0279da335c735458b0ecf134ac0c24977bc0e0c49013ef7fb50b45f5a0ce67 WatchSource:0}: Error finding container 5c0279da335c735458b0ecf134ac0c24977bc0e0c49013ef7fb50b45f5a0ce67: Status 404 returned error can't find the container with id 5c0279da335c735458b0ecf134ac0c24977bc0e0c49013ef7fb50b45f5a0ce67
Jul 08 16:35:37.351041 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:37.350994    2559 patch_prober.go:28] interesting pod/image-registry-6985974f47-dfhf7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:35:37.351232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:37.351062    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:35:37.685677 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:37.685590    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx" event={"ID":"d59253a2-e27c-4715-9306-6e7399ea8980","Type":"ContainerStarted","Data":"5c0279da335c735458b0ecf134ac0c24977bc0e0c49013ef7fb50b45f5a0ce67"}
Jul 08 16:35:40.578121 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:40.577479    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:35:41.707512 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:41.707470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9d9h" event={"ID":"ffa362c9-df89-4ef1-86f3-464642a161f4","Type":"ContainerStarted","Data":"254b05c968d2bd3d76d2dcf04c0ad46983f50fdd97098ca8d5639aa1f658e13e"}
Jul 08 16:35:41.714822 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:41.714775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx" event={"ID":"d59253a2-e27c-4715-9306-6e7399ea8980","Type":"ContainerStarted","Data":"846352d688aaf771c9dec3f4d2aa4f624ffaf901c102f7329771778ce1a212e9"}
Jul 08 16:35:41.714988 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:41.714874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx" event={"ID":"d59253a2-e27c-4715-9306-6e7399ea8980","Type":"ContainerStarted","Data":"5bf3556617d2022f09589515272233a64fb0f5121cf9854d64c76a0dae15ba3d"}
Jul 08 16:35:41.736498 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:41.736425    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-kjtvx" podStartSLOduration=2.572111331 podStartE2EDuration="6.73640657s" totalImagesPullingTime="4.164295239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:41.735539715 +0000 UTC m=+91.953736224" watchObservedRunningTime="2026-07-08 16:35:41.73640657 +0000 UTC m=+91.954603082"
Jul 08 16:35:42.722258 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:42.722207    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9d9h" event={"ID":"ffa362c9-df89-4ef1-86f3-464642a161f4","Type":"ContainerStarted","Data":"40f5486f3cde3cd3823e0936603d5259d61ce4f6f0b79db04ecb6f0e1ac0cb5c"}
Jul 08 16:35:43.305996 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.305959    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"]
Jul 08 16:35:43.309691 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.309670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.312732 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.312700    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 08 16:35:43.312885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.312865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 08 16:35:43.312989 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.312969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rjhvg\""
Jul 08 16:35:43.322854 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.322824    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"]
Jul 08 16:35:43.330916 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.330882    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k6rsq"]
Jul 08 16:35:43.335546 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.334916    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.338551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.338339    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzgdb\""
Jul 08 16:35:43.338551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.338373    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:35:43.338551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.338339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:35:43.338551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.338412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:35:43.401407 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-textfile\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401614 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995sn\" (UniqueName: \"kubernetes.io/projected/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-kube-api-access-995sn\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401614 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-root\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401614 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401543    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401614 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-metrics-client-ca\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-sys\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7197f39-e59e-45fc-9329-bff6fbaae09f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.401816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.402062 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-wtmp\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.402062 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.401933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895hf\" (UniqueName: \"kubernetes.io/projected/a7197f39-e59e-45fc-9329-bff6fbaae09f-kube-api-access-895hf\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.502788 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-textfile\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.502983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995sn\" (UniqueName: \"kubernetes.io/projected/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-kube-api-access-995sn\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.502983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-root\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.502983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.502983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-metrics-client-ca\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.502983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.502994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-sys\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:43.503077    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7197f39-e59e-45fc-9329-bff6fbaae09f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503131    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-textfile\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:43.503143    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls podName:0284cda9-24a7-49c4-b37f-5ff5fb7f34fa nodeName:}" failed. No retries permitted until 2026-07-08 16:35:44.003121019 +0000 UTC m=+94.221317505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls") pod "node-exporter-k6rsq" (UID: "0284cda9-24a7-49c4-b37f-5ff5fb7f34fa") : secret "node-exporter-tls" not found
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-wtmp\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503232 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895hf\" (UniqueName: \"kubernetes.io/projected/a7197f39-e59e-45fc-9329-bff6fbaae09f-kube-api-access-895hf\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.503615 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503594    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-root\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503712 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503689    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-sys\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503765 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503712    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503765 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503734    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-metrics-client-ca\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.503864 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:43.503774    2559 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 08 16:35:43.503864 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:35:43.503825    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls podName:a7197f39-e59e-45fc-9329-bff6fbaae09f nodeName:}" failed. No retries permitted until 2026-07-08 16:35:44.003808052 +0000 UTC m=+94.222004550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-54jr6" (UID: "a7197f39-e59e-45fc-9329-bff6fbaae09f") : secret "openshift-state-metrics-tls" not found
Jul 08 16:35:43.503864 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.503855    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-wtmp\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.504179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.504113    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7197f39-e59e-45fc-9329-bff6fbaae09f-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.506159 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.506139    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:43.506293 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.506260    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.532126 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.532090    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995sn\" (UniqueName: \"kubernetes.io/projected/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-kube-api-access-995sn\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:43.532285 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:43.532206    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895hf\" (UniqueName: \"kubernetes.io/projected/a7197f39-e59e-45fc-9329-bff6fbaae09f-kube-api-access-895hf\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:44.009126 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.009085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:44.009607 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.009148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:44.012281 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.012241    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0284cda9-24a7-49c4-b37f-5ff5fb7f34fa-node-exporter-tls\") pod \"node-exporter-k6rsq\" (UID: \"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa\") " pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:44.012476 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.012300    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7197f39-e59e-45fc-9329-bff6fbaae09f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-54jr6\" (UID: \"a7197f39-e59e-45fc-9329-bff6fbaae09f\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:44.222208 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.222167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"
Jul 08 16:35:44.249748 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:44.249672    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k6rsq"
Jul 08 16:35:45.135848 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.135736    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6"]
Jul 08 16:35:45.435680 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.435633    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-844c6d478b-hqmt6"]
Jul 08 16:35:45.443182 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.443152    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.445957 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.445930    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 08 16:35:45.446116 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.445930    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 08 16:35:45.446281 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.446240    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g579m\""
Jul 08 16:35:45.447179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.447157    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 08 16:35:45.447179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.447165    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 08 16:35:45.447336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.447164    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 08 16:35:45.447336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.447164    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-84rnl3c033ret\""
Jul 08 16:35:45.454150 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.454124    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-844c6d478b-hqmt6"]
Jul 08 16:35:45.525674 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.525885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.525885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.525885 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-grpc-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.526112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.526112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4m5x\" (UniqueName: \"kubernetes.io/projected/9a29a01a-db28-4520-9952-20c4d1c7ad02-kube-api-access-d4m5x\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.526112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.525994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.526112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.526019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a29a01a-db28-4520-9952-20c4d1c7ad02-metrics-client-ca\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627105 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627105 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4m5x\" (UniqueName: \"kubernetes.io/projected/9a29a01a-db28-4520-9952-20c4d1c7ad02-kube-api-access-d4m5x\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a29a01a-db28-4520-9952-20c4d1c7ad02-metrics-client-ca\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.627375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.627328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-grpc-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.628376 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.628317    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a29a01a-db28-4520-9952-20c4d1c7ad02-metrics-client-ca\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.630700 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.630649    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.630806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.630767    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.630886 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.630810    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-grpc-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.631220 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.631189    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-tls\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.631707 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.631678    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.634030 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.634005    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a29a01a-db28-4520-9952-20c4d1c7ad02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.637835 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.637808    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4m5x\" (UniqueName: \"kubernetes.io/projected/9a29a01a-db28-4520-9952-20c4d1c7ad02-kube-api-access-d4m5x\") pod \"thanos-querier-844c6d478b-hqmt6\" (UID: \"9a29a01a-db28-4520-9952-20c4d1c7ad02\") " pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.737721 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.737626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k6rsq" event={"ID":"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa","Type":"ContainerStarted","Data":"a460b17865497690120868a7b6c6d41d859cd7d2107e303b1dc53c4a00fbdafa"}
Jul 08 16:35:45.740324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.740292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9d9h" event={"ID":"ffa362c9-df89-4ef1-86f3-464642a161f4","Type":"ContainerStarted","Data":"5a6874f811b204d31c26fb29cdfa4186b6e03ed0160151ee2de625d1d4b347ea"}
Jul 08 16:35:45.756468 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.756431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:35:45.763833 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:45.763781    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t9d9h" podStartSLOduration=33.749119369 podStartE2EDuration="42.763760082s" totalImagesPullingTime="9.014640713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:45.762697139 +0000 UTC m=+95.980893670" watchObservedRunningTime="2026-07-08 16:35:45.763760082 +0000 UTC m=+95.981956591"
Jul 08 16:35:50.456830 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.456793    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:35:50.465336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.465307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.468438 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.468417    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:35:50.471244 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.471212    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kfjdt\""
Jul 08 16:35:50.471816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.471788    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:35:50.472599 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.472546    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:35:50.472701 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.472609    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:35:50.472701 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.472623    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:35:50.480540 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.480500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:35:50.484228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.484190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:35:50.573446 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnsf\" (UniqueName: \"kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573669 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573669 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573669 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573596    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573825 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573825 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.573825 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.573767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675186 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675186 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnsf\" (UniqueName: \"kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.675445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.675394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.676102 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.676059    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.676231 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.676138    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.676294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.676227    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.676387 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.676364    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.678640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.678608    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.678640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.678627    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.692618 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.692567    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnsf\" (UniqueName: \"kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf\") pod \"console-6d759d4cb9-t79bx\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") " pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:50.776609 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:50.776560    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:35:52.513110 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:52.513065    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7197f39_e59e_45fc_9329_bff6fbaae09f.slice/crio-74fef4557773e98b708f3024b19cdb6a555393fb7214c630d92a7409ef754c40 WatchSource:0}: Error finding container 74fef4557773e98b708f3024b19cdb6a555393fb7214c630d92a7409ef754c40: Status 404 returned error can't find the container with id 74fef4557773e98b708f3024b19cdb6a555393fb7214c630d92a7409ef754c40
Jul 08 16:35:52.686603 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.686526    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-844c6d478b-hqmt6"]
Jul 08 16:35:52.688370 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.688338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:35:52.696430 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:35:52.696405    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda270898d_0fc7_48cd_8882_a811c194897b.slice/crio-26e4b8bdbc7bb7079b8647f008c0ccb1b038f270c9375b887261b7f5d3742203 WatchSource:0}: Error finding container 26e4b8bdbc7bb7079b8647f008c0ccb1b038f270c9375b887261b7f5d3742203: Status 404 returned error can't find the container with id 26e4b8bdbc7bb7079b8647f008c0ccb1b038f270c9375b887261b7f5d3742203
Jul 08 16:35:52.774745 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.774700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"21581d97e338f9d59dc818ee3ce75bfe050df84df1dea6cc9a5780012cfa8666"}
Jul 08 16:35:52.776596 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.776553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6" event={"ID":"a7197f39-e59e-45fc-9329-bff6fbaae09f","Type":"ContainerStarted","Data":"ac11ebf14b54a5a5115664225f26b7782452ec333a23cc1ee3bdb30b4cac5db5"}
Jul 08 16:35:52.776745 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.776626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6" event={"ID":"a7197f39-e59e-45fc-9329-bff6fbaae09f","Type":"ContainerStarted","Data":"031660740fd72ad0966f58ffa042cbf2700c675ba0772183a61c9b71dff6b93d"}
Jul 08 16:35:52.776745 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.776645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6" event={"ID":"a7197f39-e59e-45fc-9329-bff6fbaae09f","Type":"ContainerStarted","Data":"74fef4557773e98b708f3024b19cdb6a555393fb7214c630d92a7409ef754c40"}
Jul 08 16:35:52.777835 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.777813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d759d4cb9-t79bx" event={"ID":"a270898d-0fc7-48cd-8882-a811c194897b","Type":"ContainerStarted","Data":"26e4b8bdbc7bb7079b8647f008c0ccb1b038f270c9375b887261b7f5d3742203"}
Jul 08 16:35:52.925271 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:52.925237    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:35:53.609375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:53.608511    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zsdxr"
Jul 08 16:35:53.787425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:53.786747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-v54bj" event={"ID":"dd42600f-0b4c-49de-9456-215bcf497926","Type":"ContainerStarted","Data":"c9fd44caf0cfceee0f36d32e01f092d07f9f511ce6cff24adac0d6f3f764ef5b"}
Jul 08 16:35:53.790088 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:53.788232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:53.800987 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:53.800955    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-v54bj"
Jul 08 16:35:53.811599 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:53.810966    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-v54bj" podStartSLOduration=2.203899651 podStartE2EDuration="23.810946949s" totalImagesPullingTime="21.607047298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:53.808943885 +0000 UTC m=+104.027140393" watchObservedRunningTime="2026-07-08 16:35:53.810946949 +0000 UTC m=+104.029143458"
Jul 08 16:35:54.802670 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:54.802625    2559 generic.go:356] "Generic (PLEG): container finished" podID="0284cda9-24a7-49c4-b37f-5ff5fb7f34fa" containerID="28bf0b077c905924dc6dda1803922f6662d73b92944fbaa4cbac6cbcf8e93164" exitCode=0
Jul 08 16:35:54.803914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:54.803880    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k6rsq" event={"ID":"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa","Type":"ContainerDied","Data":"28bf0b077c905924dc6dda1803922f6662d73b92944fbaa4cbac6cbcf8e93164"}
Jul 08 16:35:55.810865 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:55.810775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k6rsq" event={"ID":"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa","Type":"ContainerStarted","Data":"ca64a7c317d33774d6efb54889f72e0433b1bf7dc7b59b7211884117f028c8a6"}
Jul 08 16:35:55.811413 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:55.810887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k6rsq" event={"ID":"0284cda9-24a7-49c4-b37f-5ff5fb7f34fa","Type":"ContainerStarted","Data":"7a88389efca663928ffa73e7f6d8473b9d5a132165a2415e1fcdeeb0f28edaba"}
Jul 08 16:35:55.850964 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:55.850902    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k6rsq" podStartSLOduration=3.421814509 podStartE2EDuration="12.850878547s" totalImagesPullingTime="9.250255622s" totalInitContainerRuntime="178.808416ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:55.848135995 +0000 UTC m=+106.066332504" watchObservedRunningTime="2026-07-08 16:35:55.850878547 +0000 UTC m=+106.069075056"
Jul 08 16:35:58.830505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:58.830455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"bc7883dcf59d2beffa6ac98d469267cfb90ef44e2513404473acd8fec9a4d804"}
Jul 08 16:35:58.830977 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:58.830553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"8c608288442635e85faead2488af0df1c00644ba1a6819e394663889787f1f94"}
Jul 08 16:35:58.834028 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:58.833161    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6" event={"ID":"a7197f39-e59e-45fc-9329-bff6fbaae09f","Type":"ContainerStarted","Data":"7fdd42597faa35b7d6c89c1f92a895f74d8e015f631da4a44f7be729281e5e2e"}
Jul 08 16:35:58.835860 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:58.835830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d759d4cb9-t79bx" event={"ID":"a270898d-0fc7-48cd-8882-a811c194897b","Type":"ContainerStarted","Data":"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"}
Jul 08 16:35:58.856246 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:58.856190    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-54jr6" podStartSLOduration=10.417203904 podStartE2EDuration="15.856172778s" totalImagesPullingTime="5.438968874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.854883942 +0000 UTC m=+109.073080474" watchObservedRunningTime="2026-07-08 16:35:58.856172778 +0000 UTC m=+109.074369287"
Jul 08 16:35:59.852919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:35:59.852868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"a232922e1faceb099b1ba397bac5f0b03a519befb13baa357cc5cf398d0c317c"}
Jul 08 16:36:00.777746 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.777705    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:36:00.777944 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.777781    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:36:00.783732 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.783697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:36:00.806261 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.806200    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d759d4cb9-t79bx" podStartSLOduration=5.124991252 podStartE2EDuration="10.806182845s" totalImagesPullingTime="5.681191593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:58.875830709 +0000 UTC m=+109.094027219" watchObservedRunningTime="2026-07-08 16:36:00.806182845 +0000 UTC m=+111.024379418"
Jul 08 16:36:00.861714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.861609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"1c69f4b021f3319fcbd6932cba5a14da3b321bad9b14c21b50dc5859cdd7bd82"}
Jul 08 16:36:00.861714 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.861683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"ee039784276fef19eecfa4bb82020d707a3c1e51bd3507ecb8733d59e4eb0674"}
Jul 08 16:36:00.862242 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.861726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" event={"ID":"9a29a01a-db28-4520-9952-20c4d1c7ad02","Type":"ContainerStarted","Data":"bfdb1f31e0bfade9b7d1a2d44f68f088ace8c6122d86145a8da003ba1f8ff682"}
Jul 08 16:36:00.862242 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.861933    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:36:00.866841 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.866808    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:36:00.887949 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:00.887884    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6" podStartSLOduration=8.742433212 podStartE2EDuration="15.887862627s" totalImagesPullingTime="7.145429415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:00.886196583 +0000 UTC m=+111.104393105" watchObservedRunningTime="2026-07-08 16:36:00.887862627 +0000 UTC m=+111.106059138"
Jul 08 16:36:06.873201 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:06.873170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-844c6d478b-hqmt6"
Jul 08 16:36:13.916665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:13.916631    2559 generic.go:356] "Generic (PLEG): container finished" podID="610465db-6b6f-4151-ba7a-08a61e6ede7c" containerID="239f8306515fb24e1759afc82a5a172fd264c592103236a5627a2c80c0f72be7" exitCode=0
Jul 08 16:36:13.917095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:13.916686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5" event={"ID":"610465db-6b6f-4151-ba7a-08a61e6ede7c","Type":"ContainerDied","Data":"239f8306515fb24e1759afc82a5a172fd264c592103236a5627a2c80c0f72be7"}
Jul 08 16:36:13.917095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:13.917017    2559 scope.go:117] "RemoveContainer" containerID="239f8306515fb24e1759afc82a5a172fd264c592103236a5627a2c80c0f72be7"
Jul 08 16:36:14.922465 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:14.922428    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6c6v5" event={"ID":"610465db-6b6f-4151-ba7a-08a61e6ede7c","Type":"ContainerStarted","Data":"0c9189d0d0259b6454b4e133efcaf866ae00100415fdea1396015f5ef3e35ce7"}
Jul 08 16:36:17.946617 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:17.946519    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" containerName="registry" containerID="cri-o://786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2" gracePeriod=30
Jul 08 16:36:18.235099 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.235066    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:36:18.341359 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341319    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341359 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tmvd\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341380    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341446    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341484    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets\") pod \"7170501a-3805-4eb1-9475-9f838c24ba45\" (UID: \"7170501a-3805-4eb1-9475-9f838c24ba45\") "
Jul 08 16:36:18.341972 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.341799    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:18.342082 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.342024    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:18.344264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.344220    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd" (OuterVolumeSpecName: "kube-api-access-6tmvd") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "kube-api-access-6tmvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:18.344397 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.344310    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:18.344452 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.344404    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:18.344528 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.344507    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:18.344637 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.344616    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:18.350199 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.350165    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7170501a-3805-4eb1-9475-9f838c24ba45" (UID: "7170501a-3805-4eb1-9475-9f838c24ba45"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:36:18.443007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.442967    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-registry-certificates\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443003    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmvd\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-kube-api-access-6tmvd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443017    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-registry-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443030    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7170501a-3805-4eb1-9475-9f838c24ba45-bound-sa-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443043    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-image-registry-private-configuration\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443056    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7170501a-3805-4eb1-9475-9f838c24ba45-trusted-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443066    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7170501a-3805-4eb1-9475-9f838c24ba45-ca-trust-extracted\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.443260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.443077    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7170501a-3805-4eb1-9475-9f838c24ba45-installation-pull-secrets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:36:18.939473 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.939434    2559 generic.go:356] "Generic (PLEG): container finished" podID="7170501a-3805-4eb1-9475-9f838c24ba45" containerID="786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2" exitCode=0
Jul 08 16:36:18.939686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.939484    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6985974f47-dfhf7"
Jul 08 16:36:18.939686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.939521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" event={"ID":"7170501a-3805-4eb1-9475-9f838c24ba45","Type":"ContainerDied","Data":"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"}
Jul 08 16:36:18.939686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.939593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6985974f47-dfhf7" event={"ID":"7170501a-3805-4eb1-9475-9f838c24ba45","Type":"ContainerDied","Data":"38a6c97dd9c6de66dac294fba40f5b4834570d3ec2963a57614b7defdd5438a3"}
Jul 08 16:36:18.939686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.939620    2559 scope.go:117] "RemoveContainer" containerID="786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"
Jul 08 16:36:18.959196 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.959013    2559 scope.go:117] "RemoveContainer" containerID="786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"
Jul 08 16:36:18.959435 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:36:18.959348    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2\": container with ID starting with 786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2 not found: ID does not exist" containerID="786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"
Jul 08 16:36:18.959435 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.959376    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2"} err="failed to get container status \"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2\": rpc error: code = NotFound desc = could not find container \"786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2\": container with ID starting with 786e9b4f43dba9e3e73f05cd4272e0109b583b78bb0731f4a71483a17ef6b0f2 not found: ID does not exist"
Jul 08 16:36:18.963168 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.963141    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:36:18.967640 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:18.967617    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6985974f47-dfhf7"]
Jul 08 16:36:20.158761 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.158716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:36:20.161300 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.161274    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26860848-e85c-4136-9004-bb2c83bfe483-metrics-certs\") pod \"network-metrics-daemon-nb49f\" (UID: \"26860848-e85c-4136-9004-bb2c83bfe483\") " pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:36:20.243717 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.243683    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" path="/var/lib/kubelet/pods/7170501a-3805-4eb1-9475-9f838c24ba45/volumes"
Jul 08 16:36:20.458262 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.458182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g78k4\""
Jul 08 16:36:20.465033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.465006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nb49f"
Jul 08 16:36:20.634173 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.634142    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nb49f"]
Jul 08 16:36:20.636883 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:36:20.636852    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26860848_e85c_4136_9004_bb2c83bfe483.slice/crio-65f931c2180d9a99be0de29cf22bc4ca81e7694d1b44409e8ab05d2e9d9f19d4 WatchSource:0}: Error finding container 65f931c2180d9a99be0de29cf22bc4ca81e7694d1b44409e8ab05d2e9d9f19d4: Status 404 returned error can't find the container with id 65f931c2180d9a99be0de29cf22bc4ca81e7694d1b44409e8ab05d2e9d9f19d4
Jul 08 16:36:20.948718 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:20.948680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb49f" event={"ID":"26860848-e85c-4136-9004-bb2c83bfe483","Type":"ContainerStarted","Data":"65f931c2180d9a99be0de29cf22bc4ca81e7694d1b44409e8ab05d2e9d9f19d4"}
Jul 08 16:36:22.958172 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:22.958138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb49f" event={"ID":"26860848-e85c-4136-9004-bb2c83bfe483","Type":"ContainerStarted","Data":"7853081e77b6077fb554266286074b6f04c259075cb5eb3c76ace0c7b32d8806"}
Jul 08 16:36:22.958565 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:22.958190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nb49f" event={"ID":"26860848-e85c-4136-9004-bb2c83bfe483","Type":"ContainerStarted","Data":"c20d1dfcec62c72ad22893923cafd90f049a73c8fbbbef23fdff082098d679b8"}
Jul 08 16:36:22.982072 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:22.982021    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nb49f" podStartSLOduration=132.510402722 podStartE2EDuration="2m13.982005165s" totalImagesPullingTime="1.471602443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:36:22.981441983 +0000 UTC m=+133.199638512" watchObservedRunningTime="2026-07-08 16:36:22.982005165 +0000 UTC m=+133.200201675"
Jul 08 16:36:33.000171 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:33.000133    2559 generic.go:356] "Generic (PLEG): container finished" podID="62ddd43e-d605-4675-83e9-55704803d715" containerID="313ed5abcf4b8c761b6bdbf49eddba758a7c6a0d58d9407fb81de3c9c0f84977" exitCode=0
Jul 08 16:36:33.000603 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:33.000204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9" event={"ID":"62ddd43e-d605-4675-83e9-55704803d715","Type":"ContainerDied","Data":"313ed5abcf4b8c761b6bdbf49eddba758a7c6a0d58d9407fb81de3c9c0f84977"}
Jul 08 16:36:33.000603 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:33.000522    2559 scope.go:117] "RemoveContainer" containerID="313ed5abcf4b8c761b6bdbf49eddba758a7c6a0d58d9407fb81de3c9c0f84977"
Jul 08 16:36:34.005741 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:34.005705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-65gf9" event={"ID":"62ddd43e-d605-4675-83e9-55704803d715","Type":"ContainerStarted","Data":"391bff6bd28b7b885227b2d4a9940d700b43f093fe98812cbdb2f9d58135d70d"}
Jul 08 16:36:38.024868 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:38.024830    2559 generic.go:356] "Generic (PLEG): container finished" podID="7027dd97-8f1d-4c79-9429-8f9b6309bae3" containerID="b9d7e1c8638e31002b3f4446421847cf94d5e79f06d52d1ccb82a10c12102432" exitCode=0
Jul 08 16:36:38.025256 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:38.024896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt" event={"ID":"7027dd97-8f1d-4c79-9429-8f9b6309bae3","Type":"ContainerDied","Data":"b9d7e1c8638e31002b3f4446421847cf94d5e79f06d52d1ccb82a10c12102432"}
Jul 08 16:36:38.025302 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:38.025290    2559 scope.go:117] "RemoveContainer" containerID="b9d7e1c8638e31002b3f4446421847cf94d5e79f06d52d1ccb82a10c12102432"
Jul 08 16:36:39.031637 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:36:39.031597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-jcfjt" event={"ID":"7027dd97-8f1d-4c79-9429-8f9b6309bae3","Type":"ContainerStarted","Data":"4939386b11a681a568bbaf203d9fe41f8e3515308a54d1267dab7c879e99d818"}
Jul 08 16:37:12.577309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.577265    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:12.578400 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.578370    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" containerName="registry"
Jul 08 16:37:12.578400 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.578401    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" containerName="registry"
Jul 08 16:37:12.578667 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.578476    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7170501a-3805-4eb1-9475-9f838c24ba45" containerName="registry"
Jul 08 16:37:12.582768 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.582749    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.592197 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.592168    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:12.713982 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.713939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.713982 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.713986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.714218 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.714022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.714218 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.714098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.714218 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.714125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mch8k\" (UniqueName: \"kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.714218 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.714147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.714218 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.714177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815185 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815185 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mch8k\" (UniqueName: \"kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.815686 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.815442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.816092 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.816069    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.816202 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.816126    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.816202 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.816178    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.816277 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.816223    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.818071 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.818049    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.818290 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.818274    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.825689 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.825657    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mch8k\" (UniqueName: \"kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k\") pod \"console-7cb8b7dc8d-52bqw\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") " pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:12.893628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:12.893509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:13.035644 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:13.035619    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:13.038259 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:37:13.038226    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d91f90_77e8_4d76_9d72_d773020a8ced.slice/crio-e1a57232ec55b3d476cda5080e5baa80ac724dcada59da8f9dbd6a0b20631c15 WatchSource:0}: Error finding container e1a57232ec55b3d476cda5080e5baa80ac724dcada59da8f9dbd6a0b20631c15: Status 404 returned error can't find the container with id e1a57232ec55b3d476cda5080e5baa80ac724dcada59da8f9dbd6a0b20631c15
Jul 08 16:37:13.184042 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:13.183945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb8b7dc8d-52bqw" event={"ID":"80d91f90-77e8-4d76-9d72-d773020a8ced","Type":"ContainerStarted","Data":"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"}
Jul 08 16:37:13.184042 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:13.183992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb8b7dc8d-52bqw" event={"ID":"80d91f90-77e8-4d76-9d72-d773020a8ced","Type":"ContainerStarted","Data":"e1a57232ec55b3d476cda5080e5baa80ac724dcada59da8f9dbd6a0b20631c15"}
Jul 08 16:37:13.205700 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:13.205644    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cb8b7dc8d-52bqw" podStartSLOduration=1.205626036 podStartE2EDuration="1.205626036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:13.2047651 +0000 UTC m=+183.422961610" watchObservedRunningTime="2026-07-08 16:37:13.205626036 +0000 UTC m=+183.423822544"
Jul 08 16:37:22.442115 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:22.442060    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:22.894626 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:22.894557    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:33.841785 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:33.841741    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:37:47.463919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.463857    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7cb8b7dc8d-52bqw" podUID="80d91f90-77e8-4d76-9d72-d773020a8ced" containerName="console" containerID="cri-o://47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00" gracePeriod=15
Jul 08 16:37:47.709962 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.709932    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:47.720838 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.720838 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720812    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.720838 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720838    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.721075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720863    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.721075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mch8k\" (UniqueName: \"kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.721075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.721075 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.720958    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert\") pod \"80d91f90-77e8-4d76-9d72-d773020a8ced\" (UID: \"80d91f90-77e8-4d76-9d72-d773020a8ced\") "
Jul 08 16:37:47.721274 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.721228    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config" (OuterVolumeSpecName: "console-config") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:47.721274 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.721233    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:47.721375 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.721296    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca" (OuterVolumeSpecName: "service-ca") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:47.721589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.721552    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:47.723310 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.723282    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:47.723443 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.723366    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:47.723508 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.723452    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k" (OuterVolumeSpecName: "kube-api-access-mch8k") pod "80d91f90-77e8-4d76-9d72-d773020a8ced" (UID: "80d91f90-77e8-4d76-9d72-d773020a8ced"). InnerVolumeSpecName "kube-api-access-mch8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:37:47.822171 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822130    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-console-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822171 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822162    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822171 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822172    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822171 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822181    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-oauth-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822194    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mch8k\" (UniqueName: \"kubernetes.io/projected/80d91f90-77e8-4d76-9d72-d773020a8ced-kube-api-access-mch8k\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822206    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80d91f90-77e8-4d76-9d72-d773020a8ced-console-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:47.822450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:47.822218    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80d91f90-77e8-4d76-9d72-d773020a8ced-oauth-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:48.332857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.332816    2559 generic.go:356] "Generic (PLEG): container finished" podID="80d91f90-77e8-4d76-9d72-d773020a8ced" containerID="47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00" exitCode=0
Jul 08 16:37:48.333025 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.332866    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cb8b7dc8d-52bqw"
Jul 08 16:37:48.333025 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.332902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb8b7dc8d-52bqw" event={"ID":"80d91f90-77e8-4d76-9d72-d773020a8ced","Type":"ContainerDied","Data":"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"}
Jul 08 16:37:48.333025 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.332956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cb8b7dc8d-52bqw" event={"ID":"80d91f90-77e8-4d76-9d72-d773020a8ced","Type":"ContainerDied","Data":"e1a57232ec55b3d476cda5080e5baa80ac724dcada59da8f9dbd6a0b20631c15"}
Jul 08 16:37:48.333025 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.332976    2559 scope.go:117] "RemoveContainer" containerID="47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"
Jul 08 16:37:48.341704 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.341687    2559 scope.go:117] "RemoveContainer" containerID="47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"
Jul 08 16:37:48.341989 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:37:48.341969    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00\": container with ID starting with 47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00 not found: ID does not exist" containerID="47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"
Jul 08 16:37:48.342064 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.342004    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00"} err="failed to get container status \"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00\": rpc error: code = NotFound desc = could not find container \"47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00\": container with ID starting with 47bbbec4d2121209832177669b99a1d9ad91a51433779b7a8bc4f9e6f89a4e00 not found: ID does not exist"
Jul 08 16:37:48.355871 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.355845    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:48.362841 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:48.362818    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cb8b7dc8d-52bqw"]
Jul 08 16:37:50.244415 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:50.244380    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80d91f90-77e8-4d76-9d72-d773020a8ced" path="/var/lib/kubelet/pods/80d91f90-77e8-4d76-9d72-d773020a8ced/volumes"
Jul 08 16:37:58.865703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:58.865654    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d759d4cb9-t79bx" podUID="a270898d-0fc7-48cd-8882-a811c194897b" containerName="console" containerID="cri-o://4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8" gracePeriod=15
Jul 08 16:37:59.118903 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.118842    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:37:59.218131 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwnsf\" (UniqueName: \"kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218209    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218279    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218306    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218341 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config\") pod \"a270898d-0fc7-48cd-8882-a811c194897b\" (UID: \"a270898d-0fc7-48cd-8882-a811c194897b\") "
Jul 08 16:37:59.218735 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218705    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:59.218861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218730    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca" (OuterVolumeSpecName: "service-ca") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:59.218861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218845    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:59.219042 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.218890    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config" (OuterVolumeSpecName: "console-config") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:37:59.220524 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.220505    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf" (OuterVolumeSpecName: "kube-api-access-fwnsf") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "kube-api-access-fwnsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:37:59.221086 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.221056    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:59.221086 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.221067    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a270898d-0fc7-48cd-8882-a811c194897b" (UID: "a270898d-0fc7-48cd-8882-a811c194897b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:37:59.319510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319469    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-oauth-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319500    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319509    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-oauth-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319877 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319617    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319877 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319627    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a270898d-0fc7-48cd-8882-a811c194897b-console-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319877 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319636    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnsf\" (UniqueName: \"kubernetes.io/projected/a270898d-0fc7-48cd-8882-a811c194897b-kube-api-access-fwnsf\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.319877 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.319644    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a270898d-0fc7-48cd-8882-a811c194897b-console-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:37:59.384527 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.384441    2559 generic.go:356] "Generic (PLEG): container finished" podID="a270898d-0fc7-48cd-8882-a811c194897b" containerID="4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8" exitCode=0
Jul 08 16:37:59.384527 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.384487    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d759d4cb9-t79bx"
Jul 08 16:37:59.384735 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.384485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d759d4cb9-t79bx" event={"ID":"a270898d-0fc7-48cd-8882-a811c194897b","Type":"ContainerDied","Data":"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"}
Jul 08 16:37:59.384735 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.384643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d759d4cb9-t79bx" event={"ID":"a270898d-0fc7-48cd-8882-a811c194897b","Type":"ContainerDied","Data":"26e4b8bdbc7bb7079b8647f008c0ccb1b038f270c9375b887261b7f5d3742203"}
Jul 08 16:37:59.384735 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.384670    2559 scope.go:117] "RemoveContainer" containerID="4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"
Jul 08 16:37:59.394242 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.394220    2559 scope.go:117] "RemoveContainer" containerID="4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"
Jul 08 16:37:59.394653 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:37:59.394624    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8\": container with ID starting with 4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8 not found: ID does not exist" containerID="4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"
Jul 08 16:37:59.394729 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.394665    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8"} err="failed to get container status \"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8\": rpc error: code = NotFound desc = could not find container \"4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8\": container with ID starting with 4354049e08ae5a5a20813771c768c91c51ae8e3abd4e94a4c1e94dfa65a815a8 not found: ID does not exist"
Jul 08 16:37:59.409950 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.409926    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:37:59.415391 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:37:59.415367    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d759d4cb9-t79bx"]
Jul 08 16:38:00.244390 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:38:00.244356    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a270898d-0fc7-48cd-8882-a811c194897b" path="/var/lib/kubelet/pods/a270898d-0fc7-48cd-8882-a811c194897b/volumes"
Jul 08 16:39:09.788712 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.788671    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"]
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789039    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a270898d-0fc7-48cd-8882-a811c194897b" containerName="console"
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789051    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a270898d-0fc7-48cd-8882-a811c194897b" containerName="console"
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789064    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d91f90-77e8-4d76-9d72-d773020a8ced" containerName="console"
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789076    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d91f90-77e8-4d76-9d72-d773020a8ced" containerName="console"
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789170    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a270898d-0fc7-48cd-8882-a811c194897b" containerName="console"
Jul 08 16:39:09.789259 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.789181    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d91f90-77e8-4d76-9d72-d773020a8ced" containerName="console"
Jul 08 16:39:09.792206 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.792186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:09.796453 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796428    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 08 16:39:09.796634 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 08 16:39:09.796634 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796427    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6nfsb\""
Jul 08 16:39:09.796634 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 08 16:39:09.796634 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796550    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 08 16:39:09.796634 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.796443    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 08 16:39:09.802858 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.802835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"]
Jul 08 16:39:09.899128 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.899091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:09.899344 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.899134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcknr\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-kube-api-access-fcknr\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:09.899344 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:09.899253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.000138 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.000102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcknr\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-kube-api-access-fcknr\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.000334 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.000240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.000334 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.000316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.000732 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.000708    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.003111 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.003093    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-certificates\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.014318 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.014287    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcknr\" (UniqueName: \"kubernetes.io/projected/3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2-kube-api-access-fcknr\") pod \"keda-metrics-apiserver-559f75f947-hm5b6\" (UID: \"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.102515 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.102430    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:39:10.105794 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.105768    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6nfsb\""
Jul 08 16:39:10.105991 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.105896    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:39:10.113266 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.113239    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:10.118172 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.117896    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:39:10.285677 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.285641    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"]
Jul 08 16:39:10.290525 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:39:10.290495    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bec3ccc_d34a_4ff1_93e0_ddd2160d32e2.slice/crio-8f73dd6cb1a8fa1c841afd26675eb250a30b508203fed8b1855b06d79cc8431c WatchSource:0}: Error finding container 8f73dd6cb1a8fa1c841afd26675eb250a30b508203fed8b1855b06d79cc8431c: Status 404 returned error can't find the container with id 8f73dd6cb1a8fa1c841afd26675eb250a30b508203fed8b1855b06d79cc8431c
Jul 08 16:39:10.663660 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:10.663620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6" event={"ID":"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2","Type":"ContainerStarted","Data":"8f73dd6cb1a8fa1c841afd26675eb250a30b508203fed8b1855b06d79cc8431c"}
Jul 08 16:39:14.684281 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:14.684242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6" event={"ID":"3bec3ccc-d34a-4ff1-93e0-ddd2160d32e2","Type":"ContainerStarted","Data":"9d83362e324eca9b5a54e57b5d391d3a16cd3292192faab6823800bdbfa53ae2"}
Jul 08 16:39:14.684742 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:14.684385    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:39:14.706107 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:14.706047    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6" podStartSLOduration=2.003329742 podStartE2EDuration="5.706030715s" totalImagesPullingTime="3.702700973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:14.704193563 +0000 UTC m=+304.922390072" watchObservedRunningTime="2026-07-08 16:39:14.706030715 +0000 UTC m=+304.924227223"
Jul 08 16:39:25.692696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:39:25.692662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-hm5b6"
Jul 08 16:40:19.700837 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.700753    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 16:40:19.704079 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.704062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.708731 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.708706    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:40:19.709180 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.709154    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:40:19.709180 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.709168    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-wnbvh\""
Jul 08 16:40:19.709180 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.709179    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 08 16:40:19.714993 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.714970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 16:40:19.732204 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.732166    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-jkjfv"]
Jul 08 16:40:19.736056 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.736034    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.739194 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.739170    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 16:40:19.740022 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.739997    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wfwd7\""
Jul 08 16:40:19.748552 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.748521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jkjfv"]
Jul 08 16:40:19.780739 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.780699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4pv\" (UniqueName: \"kubernetes.io/projected/fd50a91b-0191-4bfe-8c64-b29c35834960-kube-api-access-vn4pv\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.780739 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.780743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fd50a91b-0191-4bfe-8c64-b29c35834960-data\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.780957 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.780817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.780957 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.780907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4qxj\" (UniqueName: \"kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.882176 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.882117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.882335 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.882208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4qxj\" (UniqueName: \"kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.882335 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.882265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4pv\" (UniqueName: \"kubernetes.io/projected/fd50a91b-0191-4bfe-8c64-b29c35834960-kube-api-access-vn4pv\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.882335 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:40:19.882290    2559 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 08 16:40:19.882335 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.882310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fd50a91b-0191-4bfe-8c64-b29c35834960-data\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.882464 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:40:19.882368    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert podName:8758a05d-4255-4769-807c-b812a1d88cc1 nodeName:}" failed. No retries permitted until 2026-07-08 16:40:20.382346925 +0000 UTC m=+370.600543411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert") pod "llmisvc-controller-manager-868c8c7759-p6tmd" (UID: "8758a05d-4255-4769-807c-b812a1d88cc1") : secret "llmisvc-webhook-server-cert" not found
Jul 08 16:40:19.882747 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.882731    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fd50a91b-0191-4bfe-8c64-b29c35834960-data\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:19.899504 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.899470    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4qxj\" (UniqueName: \"kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:19.906388 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:19.906351    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4pv\" (UniqueName: \"kubernetes.io/projected/fd50a91b-0191-4bfe-8c64-b29c35834960-kube-api-access-vn4pv\") pod \"seaweedfs-64568bcd49-jkjfv\" (UID: \"fd50a91b-0191-4bfe-8c64-b29c35834960\") " pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:20.052872 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.052827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:20.196822 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.196787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-jkjfv"]
Jul 08 16:40:20.200633 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:40:20.200604    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd50a91b_0191_4bfe_8c64_b29c35834960.slice/crio-c4cd8ff678a45b15ae73ef4c7240131a6f64e2f0c168e7fc1b37b5d64387484d WatchSource:0}: Error finding container c4cd8ff678a45b15ae73ef4c7240131a6f64e2f0c168e7fc1b37b5d64387484d: Status 404 returned error can't find the container with id c4cd8ff678a45b15ae73ef4c7240131a6f64e2f0c168e7fc1b37b5d64387484d
Jul 08 16:40:20.201901 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.201878    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:40:20.386779 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.386667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:20.389284 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.389263    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") pod \"llmisvc-controller-manager-868c8c7759-p6tmd\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") " pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:20.616378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.616344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:20.760810 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.760763    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 16:40:20.764366 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:40:20.764333    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8758a05d_4255_4769_807c_b812a1d88cc1.slice/crio-f226defc1f5327772d38712e7ca92583d6d69eea3b3f800d46a2e86795aaea0c WatchSource:0}: Error finding container f226defc1f5327772d38712e7ca92583d6d69eea3b3f800d46a2e86795aaea0c: Status 404 returned error can't find the container with id f226defc1f5327772d38712e7ca92583d6d69eea3b3f800d46a2e86795aaea0c
Jul 08 16:40:20.955696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.955601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" event={"ID":"8758a05d-4255-4769-807c-b812a1d88cc1","Type":"ContainerStarted","Data":"f226defc1f5327772d38712e7ca92583d6d69eea3b3f800d46a2e86795aaea0c"}
Jul 08 16:40:20.956677 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:20.956649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jkjfv" event={"ID":"fd50a91b-0191-4bfe-8c64-b29c35834960","Type":"ContainerStarted","Data":"c4cd8ff678a45b15ae73ef4c7240131a6f64e2f0c168e7fc1b37b5d64387484d"}
Jul 08 16:40:23.971113 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:23.971080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-jkjfv" event={"ID":"fd50a91b-0191-4bfe-8c64-b29c35834960","Type":"ContainerStarted","Data":"0bb107b134fc14a194d9b3fae2e7e6d5de7afbd8e7620ae4eab97f52c745bcfa"}
Jul 08 16:40:23.971496 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:23.971214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:23.990916 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:23.990856    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-jkjfv" podStartSLOduration=1.343743757 podStartE2EDuration="4.990833182s" totalImagesPullingTime="3.647089425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:40:23.990535346 +0000 UTC m=+374.208731856" watchObservedRunningTime="2026-07-08 16:40:23.990833182 +0000 UTC m=+374.209029692"
Jul 08 16:40:24.977174 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:24.977066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" event={"ID":"8758a05d-4255-4769-807c-b812a1d88cc1","Type":"ContainerStarted","Data":"d380146885350a4088795f68c367406e5b0cb99be0c58f567978cd99854d4eea"}
Jul 08 16:40:24.977557 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:24.977286    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:40:25.003879 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:25.003820    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" podStartSLOduration=2.160057928 podStartE2EDuration="6.003798715s" totalImagesPullingTime="3.843740787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:40:25.002166681 +0000 UTC m=+375.220363211" watchObservedRunningTime="2026-07-08 16:40:25.003798715 +0000 UTC m=+375.221995224"
Jul 08 16:40:29.979439 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:29.979404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-jkjfv"
Jul 08 16:40:55.983074 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:40:55.983044    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 16:41:17.789416 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.789377    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"]
Jul 08 16:41:17.793054 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.793030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.796723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.796692    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4vsrk\""
Jul 08 16:41:17.797734 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.797714    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 08 16:41:17.797861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.797723    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 08 16:41:17.804933 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.804903    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"]
Jul 08 16:41:17.869120 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.869077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dpct\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-kube-api-access-4dpct\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.869120 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.869127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.889764 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.889729    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4n6wf"]
Jul 08 16:41:17.893016 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.892998    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:17.895760 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.895735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jsdf8\""
Jul 08 16:41:17.908179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.908151    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4n6wf"]
Jul 08 16:41:17.969516 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.969480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dpct\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-kube-api-access-4dpct\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.969761 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.969523    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:17.969761 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.969552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.969761 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.969695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzk25\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-kube-api-access-bzk25\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:17.980706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.980673    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:17.986420 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:17.986394    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dpct\" (UniqueName: \"kubernetes.io/projected/e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a-kube-api-access-4dpct\") pod \"cert-manager-cainjector-559cdd8c87-ksdkw\" (UID: \"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:18.070894 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.070800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:18.070894 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.070883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzk25\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-kube-api-access-bzk25\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:18.081483 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.081447    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzk25\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-kube-api-access-bzk25\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:18.081674 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.081651    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8dde725-6ad6-4a16-9616-1319ad0d2350-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4n6wf\" (UID: \"b8dde725-6ad6-4a16-9616-1319ad0d2350\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:18.112560 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.112523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"
Jul 08 16:41:18.201844 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.201807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:18.262804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.262762    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw"]
Jul 08 16:41:18.267675 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:41:18.267646    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5fdfc4b_b85f_44a7_8cc1_e3cc10cf190a.slice/crio-35a8950f5b875d1f45034a4a1138357fc7d87ae5699f41c024981c8600aa05db WatchSource:0}: Error finding container 35a8950f5b875d1f45034a4a1138357fc7d87ae5699f41c024981c8600aa05db: Status 404 returned error can't find the container with id 35a8950f5b875d1f45034a4a1138357fc7d87ae5699f41c024981c8600aa05db
Jul 08 16:41:18.371225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:18.371180    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4n6wf"]
Jul 08 16:41:18.373430 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:41:18.373398    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8dde725_6ad6_4a16_9616_1319ad0d2350.slice/crio-3e5d83089735e1705414086b348b3688fd8c38f357ea090426e4d172cda3e22c WatchSource:0}: Error finding container 3e5d83089735e1705414086b348b3688fd8c38f357ea090426e4d172cda3e22c: Status 404 returned error can't find the container with id 3e5d83089735e1705414086b348b3688fd8c38f357ea090426e4d172cda3e22c
Jul 08 16:41:19.207436 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:19.207347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw" event={"ID":"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a","Type":"ContainerStarted","Data":"35a8950f5b875d1f45034a4a1138357fc7d87ae5699f41c024981c8600aa05db"}
Jul 08 16:41:19.210633 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:19.210536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf" event={"ID":"b8dde725-6ad6-4a16-9616-1319ad0d2350","Type":"ContainerStarted","Data":"3e5d83089735e1705414086b348b3688fd8c38f357ea090426e4d172cda3e22c"}
Jul 08 16:41:22.228836 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:22.228791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw" event={"ID":"e5fdfc4b-b85f-44a7-8cc1-e3cc10cf190a","Type":"ContainerStarted","Data":"3609e3dbfd6b52d497227540ad8c5a2bb7ad4db06cb68ebcf879d07bdcabcdfb"}
Jul 08 16:41:22.230354 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:22.230326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf" event={"ID":"b8dde725-6ad6-4a16-9616-1319ad0d2350","Type":"ContainerStarted","Data":"f0a6914d8dd058bb6a5b582ac13309617d11be3e1ceb5b8ab2c2ad62aa133e98"}
Jul 08 16:41:22.230467 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:22.230456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:22.255048 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:22.254985    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ksdkw" podStartSLOduration=2.108834373 podStartE2EDuration="5.254967437s" totalImagesPullingTime="3.146133064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:22.253995329 +0000 UTC m=+432.472191838" watchObservedRunningTime="2026-07-08 16:41:22.254967437 +0000 UTC m=+432.473163946"
Jul 08 16:41:22.285687 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:22.285623    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf" podStartSLOduration=2.233604319 podStartE2EDuration="5.285600446s" totalImagesPullingTime="3.051996127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:22.285167455 +0000 UTC m=+432.503363964" watchObservedRunningTime="2026-07-08 16:41:22.285600446 +0000 UTC m=+432.503796956"
Jul 08 16:41:28.236780 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:28.236746    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4n6wf"
Jul 08 16:41:51.427536 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.427500    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"]
Jul 08 16:41:51.433716 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.433690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.436477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.436451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 08 16:41:51.436477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.436464    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-qdpgf\""
Jul 08 16:41:51.444452 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.444426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"]
Jul 08 16:41:51.570033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.569994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb658bd-2735-49af-a894-31ceab2c12ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570074    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570193    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570214    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.570295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.570274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tmlt\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-kube-api-access-9tmlt\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671347 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671347 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671438    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671479    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tmlt\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-kube-api-access-9tmlt\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.671673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb658bd-2735-49af-a894-31ceab2c12ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.672026 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671856    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.672026 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.671891    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.672207 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.672177    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.672501 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.672236    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.672501 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.672432    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb658bd-2735-49af-a894-31ceab2c12ac-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.674023 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.673997    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.674126 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.674097    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.687836 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.687765    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.687969 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.687927    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tmlt\" (UniqueName: \"kubernetes.io/projected/1fb658bd-2735-49af-a894-31ceab2c12ac-kube-api-access-9tmlt\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-mbx7z\" (UID: \"1fb658bd-2735-49af-a894-31ceab2c12ac\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.744790 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.744751    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:51.895134 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:51.895102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"]
Jul 08 16:41:51.898477 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:41:51.898446    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb658bd_2735_49af_a894_31ceab2c12ac.slice/crio-928fab622646a9bf9404285b919a2a47d312f5e482a99ee8a4db631f5c3335d5 WatchSource:0}: Error finding container 928fab622646a9bf9404285b919a2a47d312f5e482a99ee8a4db631f5c3335d5: Status 404 returned error can't find the container with id 928fab622646a9bf9404285b919a2a47d312f5e482a99ee8a4db631f5c3335d5
Jul 08 16:41:52.363819 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:52.363783    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" event={"ID":"1fb658bd-2735-49af-a894-31ceab2c12ac","Type":"ContainerStarted","Data":"928fab622646a9bf9404285b919a2a47d312f5e482a99ee8a4db631f5c3335d5"}
Jul 08 16:41:55.032762 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.032720    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:41:55.033123 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.032797    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:41:55.033123 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.032828    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:41:55.380893 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.380785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" event={"ID":"1fb658bd-2735-49af-a894-31ceab2c12ac","Type":"ContainerStarted","Data":"3a06ff4404fe9434f895dc1f2c824f596a9f01f5c003a3da443cbed6e246975e"}
Jul 08 16:41:55.406649 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.406590    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podStartSLOduration=1.274606624 podStartE2EDuration="4.406556097s" totalImagesPullingTime="3.131949473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:55.405722692 +0000 UTC m=+465.623919202" watchObservedRunningTime="2026-07-08 16:41:55.406556097 +0000 UTC m=+465.624752606"
Jul 08 16:41:55.745742 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:55.745649    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:56.750092 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:56.750062    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:57.390258 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:57.390225    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:41:57.391142 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:41:57.391125    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:42:00.478762 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.478722    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"]
Jul 08 16:42:00.482272 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.482248    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.486690 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.486667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 08 16:42:00.486690 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.486667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:42:00.487953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.487934    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 08 16:42:00.487953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.487944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wmwxd\""
Jul 08 16:42:00.488158 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.487969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 08 16:42:00.492512 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.492492    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 08 16:42:00.503436 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.503411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"]
Jul 08 16:42:00.557099 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.557063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wd2l\" (UniqueName: \"kubernetes.io/projected/8929e446-21c0-4a05-8e57-e085f97b3880-kube-api-access-2wd2l\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.557296 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.557123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-metrics-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.557296 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.557209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.557388 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.557267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8929e446-21c0-4a05-8e57-e085f97b3880-manager-config\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.658179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.658137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8929e446-21c0-4a05-8e57-e085f97b3880-manager-config\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.658357 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.658199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wd2l\" (UniqueName: \"kubernetes.io/projected/8929e446-21c0-4a05-8e57-e085f97b3880-kube-api-access-2wd2l\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.658357 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.658258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-metrics-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.658357 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.658310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.658938 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.658914    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8929e446-21c0-4a05-8e57-e085f97b3880-manager-config\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.661133 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.661104    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.661233 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.661105    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8929e446-21c0-4a05-8e57-e085f97b3880-metrics-cert\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.669066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.669042    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wd2l\" (UniqueName: \"kubernetes.io/projected/8929e446-21c0-4a05-8e57-e085f97b3880-kube-api-access-2wd2l\") pod \"lws-controller-manager-6744679755-9gjmt\" (UID: \"8929e446-21c0-4a05-8e57-e085f97b3880\") " pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.792097 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.792062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:00.942967 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:00.942940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"]
Jul 08 16:42:00.945176 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:42:00.945151    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8929e446_21c0_4a05_8e57_e085f97b3880.slice/crio-2086addf8837205f7a9edf982f5214571fe9c360b70f8c4881d0eb59e8c9e212 WatchSource:0}: Error finding container 2086addf8837205f7a9edf982f5214571fe9c360b70f8c4881d0eb59e8c9e212: Status 404 returned error can't find the container with id 2086addf8837205f7a9edf982f5214571fe9c360b70f8c4881d0eb59e8c9e212
Jul 08 16:42:01.409132 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:01.409089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt" event={"ID":"8929e446-21c0-4a05-8e57-e085f97b3880","Type":"ContainerStarted","Data":"2086addf8837205f7a9edf982f5214571fe9c360b70f8c4881d0eb59e8c9e212"}
Jul 08 16:42:04.426782 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:04.426743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt" event={"ID":"8929e446-21c0-4a05-8e57-e085f97b3880","Type":"ContainerStarted","Data":"b43ae0d06c6e7d4bbf9c7e2b17f0f21e2c46508f9e58e546f840e89ec72dd2f0"}
Jul 08 16:42:04.427201 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:04.426815    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:04.449192 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:04.449135    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt" podStartSLOduration=2.024953933 podStartE2EDuration="4.449117727s" totalImagesPullingTime="2.424163794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:04.447990456 +0000 UTC m=+474.666186956" watchObservedRunningTime="2026-07-08 16:42:04.449117727 +0000 UTC m=+474.667314235"
Jul 08 16:42:15.433840 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:15.433807    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6744679755-9gjmt"
Jul 08 16:42:29.949846 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.949813    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"]
Jul 08 16:42:29.959591 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.959558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:29.971118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.971096    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 16:42:29.971274 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.971143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"]
Jul 08 16:42:29.971500 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.971467    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sthpr\""
Jul 08 16:42:29.971691 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.971677    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 16:42:29.972407 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.972394    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 08 16:42:29.975426 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:29.975408    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 08 16:42:30.114371 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.114333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmfl4\" (UniqueName: \"kubernetes.io/projected/987bd600-7105-452b-afdc-5f34714954fc-kube-api-access-kmfl4\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.114566 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.114420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/987bd600-7105-452b-afdc-5f34714954fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.114566 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.114512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/987bd600-7105-452b-afdc-5f34714954fc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.216108 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.215999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/987bd600-7105-452b-afdc-5f34714954fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.216108 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.216081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/987bd600-7105-452b-afdc-5f34714954fc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.216351 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.216119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmfl4\" (UniqueName: \"kubernetes.io/projected/987bd600-7105-452b-afdc-5f34714954fc-kube-api-access-kmfl4\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.216898 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.216872    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/987bd600-7105-452b-afdc-5f34714954fc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.218731 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.218706    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/987bd600-7105-452b-afdc-5f34714954fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.231866 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.231840    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmfl4\" (UniqueName: \"kubernetes.io/projected/987bd600-7105-452b-afdc-5f34714954fc-kube-api-access-kmfl4\") pod \"kuadrant-console-plugin-85fd4b8b9d-28qld\" (UID: \"987bd600-7105-452b-afdc-5f34714954fc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.268552 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.268523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"
Jul 08 16:42:30.423870 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.423838    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld"]
Jul 08 16:42:30.427503 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:42:30.427463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987bd600_7105_452b_afdc_5f34714954fc.slice/crio-b59b1154e25cf3aa2eb7a09c281899b034b872ecc43eed86380c4970b1fab36d WatchSource:0}: Error finding container b59b1154e25cf3aa2eb7a09c281899b034b872ecc43eed86380c4970b1fab36d: Status 404 returned error can't find the container with id b59b1154e25cf3aa2eb7a09c281899b034b872ecc43eed86380c4970b1fab36d
Jul 08 16:42:30.548825 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:30.548782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld" event={"ID":"987bd600-7105-452b-afdc-5f34714954fc","Type":"ContainerStarted","Data":"b59b1154e25cf3aa2eb7a09c281899b034b872ecc43eed86380c4970b1fab36d"}
Jul 08 16:42:36.580340 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:36.580291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld" event={"ID":"987bd600-7105-452b-afdc-5f34714954fc","Type":"ContainerStarted","Data":"790618df11927028177e5c3d5fce4f353c384407eadfe18a802ae3f43de3f9a5"}
Jul 08 16:42:36.617913 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:42:36.617863    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-28qld" podStartSLOduration=1.630616835 podStartE2EDuration="7.617845472s" totalImagesPullingTime="5.987228637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:36.617677019 +0000 UTC m=+506.835873530" watchObservedRunningTime="2026-07-08 16:42:36.617845472 +0000 UTC m=+506.836041971"
Jul 08 16:43:11.239861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.239771    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:11.263140 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.263101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:11.263327 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.263234    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:11.266275 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.266249    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9kjm9\""
Jul 08 16:43:11.382747 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.382706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knv9\" (UniqueName: \"kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9\") pod \"authorino-b6c89bb85-6smtk\" (UID: \"104e51d2-1bfc-49d1-933a-4033673e9050\") " pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:11.483673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.483632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knv9\" (UniqueName: \"kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9\") pod \"authorino-b6c89bb85-6smtk\" (UID: \"104e51d2-1bfc-49d1-933a-4033673e9050\") " pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:11.493621 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.493510    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knv9\" (UniqueName: \"kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9\") pod \"authorino-b6c89bb85-6smtk\" (UID: \"104e51d2-1bfc-49d1-933a-4033673e9050\") " pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:11.572542 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.572493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:11.715635 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.715607    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:11.717855 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:43:11.717825    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod104e51d2_1bfc_49d1_933a_4033673e9050.slice/crio-b364216a3499f156fc10f728d7a566c83097135ea2315f0010b9c7270e0bcd64 WatchSource:0}: Error finding container b364216a3499f156fc10f728d7a566c83097135ea2315f0010b9c7270e0bcd64: Status 404 returned error can't find the container with id b364216a3499f156fc10f728d7a566c83097135ea2315f0010b9c7270e0bcd64
Jul 08 16:43:11.742527 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:11.742496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-6smtk" event={"ID":"104e51d2-1bfc-49d1-933a-4033673e9050","Type":"ContainerStarted","Data":"b364216a3499f156fc10f728d7a566c83097135ea2315f0010b9c7270e0bcd64"}
Jul 08 16:43:15.764649 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:15.764606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-6smtk" event={"ID":"104e51d2-1bfc-49d1-933a-4033673e9050","Type":"ContainerStarted","Data":"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"}
Jul 08 16:43:15.785309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:15.785252    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-6smtk" podStartSLOduration=0.965162907 podStartE2EDuration="4.785235472s" totalImagesPullingTime="3.820072565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:15.784161169 +0000 UTC m=+546.002357679" watchObservedRunningTime="2026-07-08 16:43:15.785235472 +0000 UTC m=+546.003431981"
Jul 08 16:43:17.500318 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:17.500280    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:17.774699 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:17.774660    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-6smtk" podUID="104e51d2-1bfc-49d1-933a-4033673e9050" containerName="authorino" containerID="cri-o://fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd" gracePeriod=30
Jul 08 16:43:18.036479 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.036402    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:18.142207 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.142173    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7knv9\" (UniqueName: \"kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9\") pod \"104e51d2-1bfc-49d1-933a-4033673e9050\" (UID: \"104e51d2-1bfc-49d1-933a-4033673e9050\") "
Jul 08 16:43:18.144645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.144616    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9" (OuterVolumeSpecName: "kube-api-access-7knv9") pod "104e51d2-1bfc-49d1-933a-4033673e9050" (UID: "104e51d2-1bfc-49d1-933a-4033673e9050"). InnerVolumeSpecName "kube-api-access-7knv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:18.243051 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.243023    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7knv9\" (UniqueName: \"kubernetes.io/projected/104e51d2-1bfc-49d1-933a-4033673e9050-kube-api-access-7knv9\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:43:18.780935 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.780902    2559 generic.go:356] "Generic (PLEG): container finished" podID="104e51d2-1bfc-49d1-933a-4033673e9050" containerID="fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd" exitCode=0
Jul 08 16:43:18.781383 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.780945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-6smtk" event={"ID":"104e51d2-1bfc-49d1-933a-4033673e9050","Type":"ContainerDied","Data":"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"}
Jul 08 16:43:18.781383 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.780951    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-6smtk"
Jul 08 16:43:18.781383 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.780973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-6smtk" event={"ID":"104e51d2-1bfc-49d1-933a-4033673e9050","Type":"ContainerDied","Data":"b364216a3499f156fc10f728d7a566c83097135ea2315f0010b9c7270e0bcd64"}
Jul 08 16:43:18.781383 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.780991    2559 scope.go:117] "RemoveContainer" containerID="fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"
Jul 08 16:43:18.789805 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.789785    2559 scope.go:117] "RemoveContainer" containerID="fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"
Jul 08 16:43:18.790078 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:43:18.790056    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd\": container with ID starting with fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd not found: ID does not exist" containerID="fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"
Jul 08 16:43:18.790139 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.790088    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd"} err="failed to get container status \"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd\": rpc error: code = NotFound desc = could not find container \"fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd\": container with ID starting with fb4cf1f56cdda1a45c962ab7d14cf5be8813fa4b68ab42c7245b70466718accd not found: ID does not exist"
Jul 08 16:43:18.805654 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.805627    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:18.813357 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:18.813331    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-6smtk"]
Jul 08 16:43:20.251414 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:43:20.251383    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104e51d2-1bfc-49d1-933a-4033673e9050" path="/var/lib/kubelet/pods/104e51d2-1bfc-49d1-933a-4033673e9050/volumes"
Jul 08 16:44:10.136046 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:10.136015    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:44:10.136648 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:10.136440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:44:32.778484 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.778398    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mc5rq"]
Jul 08 16:44:32.780953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.778812    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104e51d2-1bfc-49d1-933a-4033673e9050" containerName="authorino"
Jul 08 16:44:32.780953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.778825    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="104e51d2-1bfc-49d1-933a-4033673e9050" containerName="authorino"
Jul 08 16:44:32.780953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.778885    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="104e51d2-1bfc-49d1-933a-4033673e9050" containerName="authorino"
Jul 08 16:44:32.781932 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.781910    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mc5rq"
Jul 08 16:44:32.793414 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.793387    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mc5rq"]
Jul 08 16:44:32.893531 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.893492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:32.893531 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.893537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnhc\" (UniqueName: \"kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:32.994481 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.994433    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:32.994481 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.994485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnhc\" (UniqueName: \"kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:32.994855 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:32.994835    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:33.007377 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:33.007348    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnhc\" (UniqueName: \"kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc\") pod \"s3-init-mc5rq\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") " pod="kserve/s3-init-mc5rq"
Jul 08 16:44:33.090643 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:33.090547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mc5rq"
Jul 08 16:44:33.233546 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:33.233510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mc5rq"]
Jul 08 16:44:33.236906 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:44:33.236865    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdd86e2b_679c_4dba_9e80_9d53809c0b5e.slice/crio-a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1 WatchSource:0}: Error finding container a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1: Status 404 returned error can't find the container with id a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1
Jul 08 16:44:34.134539 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:34.134505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerStarted","Data":"a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1"}
Jul 08 16:44:38.156916 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:38.156870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerStarted","Data":"644565bef1661b6209d0de27231784f9475a3ef9549a752a42ca06d972994d24"}
Jul 08 16:44:40.169808 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:40.169773    2559 generic.go:356] "Generic (PLEG): container finished" podID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerID="644565bef1661b6209d0de27231784f9475a3ef9549a752a42ca06d972994d24" exitCode=0
Jul 08 16:44:40.170220 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:40.169839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerDied","Data":"644565bef1661b6209d0de27231784f9475a3ef9549a752a42ca06d972994d24"}
Jul 08 16:44:45.204469 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:45.204435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerStarted","Data":"abcb39a472b58d0a9985292d9cb1ef2aee48401b492bec0b6d2bd6d109a25b7b"}
Jul 08 16:44:45.226331 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:44:45.226258    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mc5rq" podStartSLOduration=2.017135363 podStartE2EDuration="13.226240627s" totalImagesPullingTime="8.967868511s" totalInitContainerRuntime="2.241236753s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:45.225739837 +0000 UTC m=+635.443936427" watchObservedRunningTime="2026-07-08 16:44:45.226240627 +0000 UTC m=+635.444437137"
Jul 08 16:45:50.519537 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:50.519500    2559 generic.go:356] "Generic (PLEG): container finished" podID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerID="abcb39a472b58d0a9985292d9cb1ef2aee48401b492bec0b6d2bd6d109a25b7b" exitCode=0
Jul 08 16:45:50.520085 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:50.519602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerDied","Data":"abcb39a472b58d0a9985292d9cb1ef2aee48401b492bec0b6d2bd6d109a25b7b"}
Jul 08 16:45:51.657187 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.657161    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mc5rq"
Jul 08 16:45:51.778612 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.778491    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qnhc\" (UniqueName: \"kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc\") pod \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") "
Jul 08 16:45:51.778612 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.778549    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache\") pod \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\" (UID: \"fdd86e2b-679c-4dba-9e80-9d53809c0b5e\") "
Jul 08 16:45:51.781008 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.780967    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc" (OuterVolumeSpecName: "kube-api-access-5qnhc") pod "fdd86e2b-679c-4dba-9e80-9d53809c0b5e" (UID: "fdd86e2b-679c-4dba-9e80-9d53809c0b5e"). InnerVolumeSpecName "kube-api-access-5qnhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:45:51.797628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.797600    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdd86e2b-679c-4dba-9e80-9d53809c0b5e" (UID: "fdd86e2b-679c-4dba-9e80-9d53809c0b5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:45:51.879795 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.879765    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:45:51.879795 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:51.879793    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnhc\" (UniqueName: \"kubernetes.io/projected/fdd86e2b-679c-4dba-9e80-9d53809c0b5e-kube-api-access-5qnhc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:45:52.530849 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:52.530816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mc5rq" event={"ID":"fdd86e2b-679c-4dba-9e80-9d53809c0b5e","Type":"ContainerDied","Data":"a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1"}
Jul 08 16:45:52.530849 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:52.530844    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mc5rq"
Jul 08 16:45:52.530849 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:45:52.530854    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9652c1bba3312483ab09fa9f3c533cbc4633146863d39e89b6dc882789b4eb1"
Jul 08 16:46:05.539812 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.539775    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:05.540228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.540128    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerName="s3-init"
Jul 08 16:46:05.540228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.540139    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerName="s3-init"
Jul 08 16:46:05.540228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.540147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerName="download-hf-model"
Jul 08 16:46:05.540228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.540152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerName="download-hf-model"
Jul 08 16:46:05.540228 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.540208    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd86e2b-679c-4dba-9e80-9d53809c0b5e" containerName="s3-init"
Jul 08 16:46:05.543233 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.543217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.547639 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.547617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:46:05.547835 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.547820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-dc4rb\""
Jul 08 16:46:05.548839 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.548820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 08 16:46:05.548949 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.548872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:46:05.562550 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.562513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:05.700676 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700648    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700676 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9jg\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700732    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.700931 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.701145 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.700945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.801921 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.801817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.801921 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.801873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.801921 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.801915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.801938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.801978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9jg\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802225 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802348    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802417    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802647 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802522    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802708 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802673    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.802856 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.802838    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.804692 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.804668    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.804692 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.804688    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.817345 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.817315    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.819149 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.819115    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9jg\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gb4wz\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:05.853664 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:05.853628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:06.011672 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.011643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:06.013743 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:46:06.013715    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfe4811_b058_4cdf_bf32_a2b6ff3bf0b1.slice/crio-1c94366ef89eb5bdb375f77a38a5f72623ad18ef86656e1048d1bdfff872e05c WatchSource:0}: Error finding container 1c94366ef89eb5bdb375f77a38a5f72623ad18ef86656e1048d1bdfff872e05c: Status 404 returned error can't find the container with id 1c94366ef89eb5bdb375f77a38a5f72623ad18ef86656e1048d1bdfff872e05c
Jul 08 16:46:06.015766 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.015725    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:06.015879 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.015809    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:06.015879 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.015858    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:06.597659 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.597613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" event={"ID":"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1","Type":"ContainerStarted","Data":"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"}
Jul 08 16:46:06.598182 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.597679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" event={"ID":"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1","Type":"ContainerStarted","Data":"1c94366ef89eb5bdb375f77a38a5f72623ad18ef86656e1048d1bdfff872e05c"}
Jul 08 16:46:06.637534 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.637475    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" podStartSLOduration=1.637460098 podStartE2EDuration="1.637460098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:06.636898277 +0000 UTC m=+716.855094785" watchObservedRunningTime="2026-07-08 16:46:06.637460098 +0000 UTC m=+716.855656606"
Jul 08 16:46:06.854292 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:06.854193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:07.859749 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:07.859713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:08.606607 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:08.606582    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:08.607425 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:08.607406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:15.688394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.688356    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:15.692562 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.692536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.699799 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.699774    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 08 16:46:15.699934 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.699810    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:46:15.708505 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.708479    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:15.798510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbqvf\" (UniqueName: \"kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.798510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.798804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.798804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.798804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.798804 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.798752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900054 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbqvf\" (UniqueName: \"kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900544 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900503    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900730 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900712    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.900778 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.900736    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.902720 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.902692    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.902957 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.902940    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:15.911538 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:15.911502    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbqvf\" (UniqueName: \"kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:16.003665 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:16.003527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:16.151511 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:16.151451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:16.154130 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:46:16.154100    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14a7274_03b1_4db8_b906_407b340236e9.slice/crio-5dc5b7ba22b1506cd995d97714b5c85292fa3717196994bfb72c446c058a6099 WatchSource:0}: Error finding container 5dc5b7ba22b1506cd995d97714b5c85292fa3717196994bfb72c446c058a6099: Status 404 returned error can't find the container with id 5dc5b7ba22b1506cd995d97714b5c85292fa3717196994bfb72c446c058a6099
Jul 08 16:46:16.156348 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:16.156331    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:46:16.648428 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:16.648389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" event={"ID":"d14a7274-03b1-4db8-b906-407b340236e9","Type":"ContainerStarted","Data":"5dc5b7ba22b1506cd995d97714b5c85292fa3717196994bfb72c446c058a6099"}
Jul 08 16:46:17.656635 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:17.656602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" event={"ID":"d14a7274-03b1-4db8-b906-407b340236e9","Type":"ContainerStarted","Data":"c5beda8ba6e6a1af46e183512d946fa38b45e030556050baa98943562e6ecc12"}
Jul 08 16:46:17.682432 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:17.682377    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" podStartSLOduration=1.298419337 podStartE2EDuration="2.682359239s" totalImagesPullingTime="1.383939902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:17.680314474 +0000 UTC m=+727.898510985" watchObservedRunningTime="2026-07-08 16:46:17.682359239 +0000 UTC m=+727.900555788"
Jul 08 16:46:18.003785 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:18.003670    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:20.788256 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:20.788212    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:20.788856 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:20.788500    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerName="istio-proxy" containerID="cri-o://67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9" gracePeriod=30
Jul 08 16:46:24.647788 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:24.647741    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 08 16:46:25.242513 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.242488    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:25.301835 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.301801    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.301871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.301926    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.301964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9jg\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.301988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302068 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302061    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302373 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302127    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302373 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302126    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:25.302373 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302178    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token\") pod \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\" (UID: \"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1\") "
Jul 08 16:46:25.302514 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302397    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:25.302589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302523    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.302589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302542    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.302589 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302549    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:25.302751 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302624    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:25.302751 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.302635    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:25.304771 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.304739    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:25.304771 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.304740    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg" (OuterVolumeSpecName: "kube-api-access-rm9jg") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "kube-api-access-rm9jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:25.304954 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.304842    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:25.304954 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.304855    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" (UID: "8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 16:46:25.403445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403411    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403445 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403443    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403456    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403469    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403482    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9jg\" (UniqueName: \"kubernetes.io/projected/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-kube-api-access-rm9jg\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403521    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.403725 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.403533    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:25.696890 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.696793    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerID="67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9" exitCode=0
Jul 08 16:46:25.696890 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.696838    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"
Jul 08 16:46:25.697378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.696883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" event={"ID":"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1","Type":"ContainerDied","Data":"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"}
Jul 08 16:46:25.697378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.696954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz" event={"ID":"8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1","Type":"ContainerDied","Data":"1c94366ef89eb5bdb375f77a38a5f72623ad18ef86656e1048d1bdfff872e05c"}
Jul 08 16:46:25.697378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.696975    2559 scope.go:117] "RemoveContainer" containerID="67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"
Jul 08 16:46:25.706398 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.706374    2559 scope.go:117] "RemoveContainer" containerID="67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"
Jul 08 16:46:25.706694 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:46:25.706675    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9\": container with ID starting with 67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9 not found: ID does not exist" containerID="67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"
Jul 08 16:46:25.706767 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.706703    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9"} err="failed to get container status \"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9\": rpc error: code = NotFound desc = could not find container \"67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9\": container with ID starting with 67e1cc008abd64567be38220d01589f49e19cabaf2ee1611d55bbc03424535a9 not found: ID does not exist"
Jul 08 16:46:25.723962 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.723934    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:25.729662 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:25.729634    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gb4wz"]
Jul 08 16:46:26.004129 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:26.004037    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:26.017203 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:26.017170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:26.244835 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:26.244799    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" path="/var/lib/kubelet/pods/8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1/volumes"
Jul 08 16:46:26.714106 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:26.714069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:36.173651 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.173614    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:46:36.174062 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.174003    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerName="istio-proxy"
Jul 08 16:46:36.174062 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.174015    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerName="istio-proxy"
Jul 08 16:46:36.174146 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.174071    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfe4811-b058-4cdf-bf32-a2b6ff3bf0b1" containerName="istio-proxy"
Jul 08 16:46:36.177226 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.177200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.179724 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.179701    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:46:36.179846 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.179791    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 08 16:46:36.191272 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.191250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:46:36.299716 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299677    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299716 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvt67\" (UniqueName: \"kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.299941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.299908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.400814 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.400814 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvt67\" (UniqueName: \"kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401066 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.400970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.401248    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401349 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.401298    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401395 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.401370    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.401449 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.401430    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.403426 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.403402    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.403595 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.403557    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.411496 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.411474    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvt67\" (UniqueName: \"kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67\") pod \"precise-prefix-cache-test-kserve-595cd9479-68wpf\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.487673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.487549    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:36.564114 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.564085    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:46:36.569166 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.569143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.573236 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.573206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-v6w78\""
Jul 08 16:46:36.583058 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.583025    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:46:36.654273 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.654249    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:46:36.656846 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:46:36.656817    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24863edf_a2eb_4a0a_8453_c8dd91d6dbe2.slice/crio-1fe52bcac2f6603e57b45e6d67256b4a6f65460a2bb106e48607e5cc8075288d WatchSource:0}: Error finding container 1fe52bcac2f6603e57b45e6d67256b4a6f65460a2bb106e48607e5cc8075288d: Status 404 returned error can't find the container with id 1fe52bcac2f6603e57b45e6d67256b4a6f65460a2bb106e48607e5cc8075288d
Jul 08 16:46:36.703910 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.703873    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.704104 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.703934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.704104 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.703973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.704104 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.704033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.704104 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.704075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtp7\" (UniqueName: \"kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.704309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.704126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.757637 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.757522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerStarted","Data":"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"}
Jul 08 16:46:36.757637 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.757614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerStarted","Data":"1fe52bcac2f6603e57b45e6d67256b4a6f65460a2bb106e48607e5cc8075288d"}
Jul 08 16:46:36.805096 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805285 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805285 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805285 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805437 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805437 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtp7\" (UniqueName: \"kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805615 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805555    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805694 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805600    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.805932 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805904    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.806042 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.805951    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.808090 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.808069    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.816657 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.816619    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtp7\" (UniqueName: \"kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7\") pod \"precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:36.881678 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:36.881642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:46:37.037324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:37.037299    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:46:37.039363 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:46:37.039333    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a42218e_b808_4371_a6a0_fe4ce054ca0a.slice/crio-ffbd70ceb7023cafe53332514a0b17a232dbab703830f89bf82cc15b339b4b81 WatchSource:0}: Error finding container ffbd70ceb7023cafe53332514a0b17a232dbab703830f89bf82cc15b339b4b81: Status 404 returned error can't find the container with id ffbd70ceb7023cafe53332514a0b17a232dbab703830f89bf82cc15b339b4b81
Jul 08 16:46:37.768411 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:37.768240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerStarted","Data":"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"}
Jul 08 16:46:37.768411 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:37.768312    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerStarted","Data":"ffbd70ceb7023cafe53332514a0b17a232dbab703830f89bf82cc15b339b4b81"}
Jul 08 16:46:38.775688 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:38.775647    2559 generic.go:356] "Generic (PLEG): container finished" podID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerID="853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec" exitCode=0
Jul 08 16:46:38.776519 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:38.775728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerDied","Data":"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"}
Jul 08 16:46:39.493284 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:39.493220    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:40.793313 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:40.793275    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerStarted","Data":"daf6e01e70b16c0af1d22f78828a567be065755f194bb567e4f2fe60fed6e9e3"}
Jul 08 16:46:43.839118 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:43.839081    2559 generic.go:356] "Generic (PLEG): container finished" podID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerID="2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef" exitCode=0
Jul 08 16:46:43.839605 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:43.839146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerDied","Data":"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"}
Jul 08 16:46:44.848151 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:44.848115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerStarted","Data":"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"}
Jul 08 16:46:44.875730 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:44.875645    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" podStartSLOduration=2.188634251 podStartE2EDuration="8.875618885s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.686984634s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:44.872020069 +0000 UTC m=+755.090216590" watchObservedRunningTime="2026-07-08 16:46:44.875618885 +0000 UTC m=+755.093815397"
Jul 08 16:46:45.487908 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:45.487863    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:46.487892 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:46.487846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:46.504144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:46.504103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:46.876310 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:46.876279    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:46:48.874891 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.874843    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerID="3a06ff4404fe9434f895dc1f2c824f596a9f01f5c003a3da443cbed6e246975e" exitCode=0
Jul 08 16:46:48.875368 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.875029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" event={"ID":"1fb658bd-2735-49af-a894-31ceab2c12ac","Type":"ContainerDied","Data":"3a06ff4404fe9434f895dc1f2c824f596a9f01f5c003a3da443cbed6e246975e"}
Jul 08 16:46:48.875663 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.875633    2559 scope.go:117] "RemoveContainer" containerID="3a06ff4404fe9434f895dc1f2c824f596a9f01f5c003a3da443cbed6e246975e"
Jul 08 16:46:48.876734 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.876702    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:48.876831 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.876797    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:48.876893 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:48.876845    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 16:46:49.882668 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:49.882635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" event={"ID":"1fb658bd-2735-49af-a894-31ceab2c12ac","Type":"ContainerStarted","Data":"1ee2c42163fc2be7a9468ac6f600d30570ec747b9aae7687a69085d3463498d1"}
Jul 08 16:46:50.745105 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.745069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:46:50.745462 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.745438    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Jul 08 16:46:50.745542 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.745490    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podUID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 08 16:46:50.889874 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.889787    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6_9a42218e-b808-4371-a6a0-fe4ce054ca0a/main/0.log"
Jul 08 16:46:50.890378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.890283    2559 generic.go:356] "Generic (PLEG): container finished" podID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerID="daf6e01e70b16c0af1d22f78828a567be065755f194bb567e4f2fe60fed6e9e3" exitCode=1
Jul 08 16:46:50.890378 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:50.890356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerDied","Data":"daf6e01e70b16c0af1d22f78828a567be065755f194bb567e4f2fe60fed6e9e3"}
Jul 08 16:46:51.746168 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:51.746115    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Jul 08 16:46:51.746380 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:51.746200    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podUID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 08 16:46:52.745384 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:52.745311    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Jul 08 16:46:52.746026 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:52.745381    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podUID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 08 16:46:53.746072 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:53.746028    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Jul 08 16:46:53.746614 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:53.746101    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podUID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 08 16:46:54.537117 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.537065    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" podUID="d14a7274-03b1-4db8-b906-407b340236e9" containerName="main" containerID="cri-o://c5beda8ba6e6a1af46e183512d946fa38b45e030556050baa98943562e6ecc12" gracePeriod=45
Jul 08 16:46:54.707604 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.703666    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" podUID="d14a7274-03b1-4db8-b906-407b340236e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 08 16:46:54.745210 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.745169    2559 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Jul 08 16:46:54.745380 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.745232    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z" podUID="1fb658bd-2735-49af-a894-31ceab2c12ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Jul 08 16:46:54.939442 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.939344    2559 generic.go:356] "Generic (PLEG): container finished" podID="d14a7274-03b1-4db8-b906-407b340236e9" containerID="c5beda8ba6e6a1af46e183512d946fa38b45e030556050baa98943562e6ecc12" exitCode=0
Jul 08 16:46:54.939840 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:54.939496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" event={"ID":"d14a7274-03b1-4db8-b906-407b340236e9","Type":"ContainerDied","Data":"c5beda8ba6e6a1af46e183512d946fa38b45e030556050baa98943562e6ecc12"}
Jul 08 16:46:55.364179 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.364129    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:55.502919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.502830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.502919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.502878    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.502919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.502910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.503193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.502933    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.503193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.503193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503058    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbqvf\" (UniqueName: \"kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf\") pod \"d14a7274-03b1-4db8-b906-407b340236e9\" (UID: \"d14a7274-03b1-4db8-b906-407b340236e9\") "
Jul 08 16:46:55.503193 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503151    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home" (OuterVolumeSpecName: "home") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:55.503447 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503426    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.503510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503445    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:55.503510 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.503459    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:55.506110 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.506066    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:46:55.506234 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.506163    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf" (OuterVolumeSpecName: "kube-api-access-bbqvf") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "kube-api-access-bbqvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:55.506420 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.506382    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm" (OuterVolumeSpecName: "dshm") pod "d14a7274-03b1-4db8-b906-407b340236e9" (UID: "d14a7274-03b1-4db8-b906-407b340236e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:55.604652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.604610    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.604652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.604650    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.604652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.604663    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d14a7274-03b1-4db8-b906-407b340236e9-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.604900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.604674    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14a7274-03b1-4db8-b906-407b340236e9-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.604900 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.604686    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbqvf\" (UniqueName: \"kubernetes.io/projected/d14a7274-03b1-4db8-b906-407b340236e9-kube-api-access-bbqvf\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:46:55.750473 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.750428    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:46:55.750805 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.750785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:46:55.751664 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.751638    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-mbx7z"
Jul 08 16:46:55.947400 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.947369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"
Jul 08 16:46:55.947912 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.947399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch" event={"ID":"d14a7274-03b1-4db8-b906-407b340236e9","Type":"ContainerDied","Data":"5dc5b7ba22b1506cd995d97714b5c85292fa3717196994bfb72c446c058a6099"}
Jul 08 16:46:55.947912 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.947477    2559 scope.go:117] "RemoveContainer" containerID="c5beda8ba6e6a1af46e183512d946fa38b45e030556050baa98943562e6ecc12"
Jul 08 16:46:55.979280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.979245    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:55.984723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:55.984692    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc526tch"]
Jul 08 16:46:56.247913 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:46:56.247809    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d14a7274-03b1-4db8-b906-407b340236e9" path="/var/lib/kubelet/pods/d14a7274-03b1-4db8-b906-407b340236e9/volumes"
Jul 08 16:47:14.059250 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:14.059205    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6_9a42218e-b808-4371-a6a0-fe4ce054ca0a/main/0.log"
Jul 08 16:47:14.059764 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:14.059620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerStarted","Data":"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"}
Jul 08 16:47:14.059966 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:14.059953    2559 scope.go:117] "RemoveContainer" containerID="daf6e01e70b16c0af1d22f78828a567be065755f194bb567e4f2fe60fed6e9e3"
Jul 08 16:47:15.069196 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:15.069160    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6_9a42218e-b808-4371-a6a0-fe4ce054ca0a/main/0.log"
Jul 08 16:47:15.069712 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:15.069562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerStarted","Data":"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"}
Jul 08 16:47:15.070592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:15.070556    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:15.117602 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:15.117139    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" podStartSLOduration=3.466062086 podStartE2EDuration="39.117115478s" totalImagesPullingTime="34.950440179s" totalInitContainerRuntime="700.613213ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:15.112920971 +0000 UTC m=+785.331117483" watchObservedRunningTime="2026-07-08 16:47:15.117115478 +0000 UTC m=+785.335311990"
Jul 08 16:47:16.882450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:16.882414    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:16.882867 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:16.882490    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:26.883477 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:47:26.883432    2559 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.39:9003", ServerName: "10.132.0.39:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.39:9003: connect: connection refused"
Jul 08 16:47:26.885099 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:26.885081    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:26.886504 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:26.886483    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:27.882598 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:27.882534    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.39:9003\" within 1s: context deadline exceeded"
Jul 08 16:47:47.132388 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:47.132302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:53.718071 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:53.718033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:47:53.723254 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:53.723221    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:47:53.723528 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:53.723501    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="tokenizer" containerID="cri-o://22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3" gracePeriod=60
Jul 08 16:47:53.733346 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:47:53.733306    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:47:53.733346 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:47:53.733346 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:47:53.733346 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8" cmd=["/bin/sleep","15"]
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:53.733367    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" message="" err=<
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:47:53.733382    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main" containerID="cri-o://29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"
Jul 08 16:47:53.733673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:53.733421    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main" containerID="cri-o://29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8" gracePeriod=60
Jul 08 16:47:54.268865 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:54.268830    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6_9a42218e-b808-4371-a6a0-fe4ce054ca0a/main/0.log"
Jul 08 16:47:54.269271 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:54.269245    2559 generic.go:356] "Generic (PLEG): container finished" podID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerID="29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8" exitCode=0
Jul 08 16:47:54.269344 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:54.269321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerDied","Data":"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"}
Jul 08 16:47:54.269400 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:54.269390    2559 scope.go:117] "RemoveContainer" containerID="daf6e01e70b16c0af1d22f78828a567be065755f194bb567e4f2fe60fed6e9e3"
Jul 08 16:47:55.485757 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.485730    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:55.681718 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.681718 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681695    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.681983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681740    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmtp7\" (UniqueName: \"kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.681983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681764    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.681983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681832    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.681983 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.681872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location\") pod \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\" (UID: \"9a42218e-b808-4371-a6a0-fe4ce054ca0a\") "
Jul 08 16:47:55.682655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682061    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:55.682655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682079    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:55.682655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682240    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-tmp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:55.682655 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682256    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:55.682910 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682745    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:55.682910 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.682810    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:55.684072 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.684051    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:47:55.684770 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.684750    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7" (OuterVolumeSpecName: "kube-api-access-kmtp7") pod "9a42218e-b808-4371-a6a0-fe4ce054ca0a" (UID: "9a42218e-b808-4371-a6a0-fe4ce054ca0a"). InnerVolumeSpecName "kube-api-access-kmtp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:47:55.783277 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.783244    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtp7\" (UniqueName: \"kubernetes.io/projected/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kube-api-access-kmtp7\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:55.783277 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.783274    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tokenizer-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:55.783277 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.783284    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a42218e-b808-4371-a6a0-fe4ce054ca0a-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:55.783513 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:55.783294    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a42218e-b808-4371-a6a0-fe4ce054ca0a-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:47:56.285851 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.285818    2559 generic.go:356] "Generic (PLEG): container finished" podID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerID="22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3" exitCode=0
Jul 08 16:47:56.286033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.285875    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"
Jul 08 16:47:56.286033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.285904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerDied","Data":"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"}
Jul 08 16:47:56.286033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.285979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6" event={"ID":"9a42218e-b808-4371-a6a0-fe4ce054ca0a","Type":"ContainerDied","Data":"ffbd70ceb7023cafe53332514a0b17a232dbab703830f89bf82cc15b339b4b81"}
Jul 08 16:47:56.286033 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.286008    2559 scope.go:117] "RemoveContainer" containerID="29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"
Jul 08 16:47:56.295390 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.295369    2559 scope.go:117] "RemoveContainer" containerID="22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"
Jul 08 16:47:56.303814 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.303795    2559 scope.go:117] "RemoveContainer" containerID="853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"
Jul 08 16:47:56.313787 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.312141    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:47:56.316816 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.316789    2559 scope.go:117] "RemoveContainer" containerID="29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"
Jul 08 16:47:56.317209 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:47:56.317186    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8\": container with ID starting with 29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8 not found: ID does not exist" containerID="29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"
Jul 08 16:47:56.317307 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.317215    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8"} err="failed to get container status \"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8\": rpc error: code = NotFound desc = could not find container \"29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8\": container with ID starting with 29cd05eb8c6d8405fd9611c01ddfd9a546c7bbce446475a088c5f913b35494a8 not found: ID does not exist"
Jul 08 16:47:56.317307 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.317235    2559 scope.go:117] "RemoveContainer" containerID="22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"
Jul 08 16:47:56.317518 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:47:56.317498    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3\": container with ID starting with 22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3 not found: ID does not exist" containerID="22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"
Jul 08 16:47:56.317648 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.317529    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3"} err="failed to get container status \"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3\": rpc error: code = NotFound desc = could not find container \"22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3\": container with ID starting with 22ea68b88cf002478f3d4e845907d9178d4b383331fa52574f1631a6a466d9c3 not found: ID does not exist"
Jul 08 16:47:56.317648 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.317555    2559 scope.go:117] "RemoveContainer" containerID="853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"
Jul 08 16:47:56.317864 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:47:56.317847    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec\": container with ID starting with 853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec not found: ID does not exist" containerID="853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"
Jul 08 16:47:56.317903 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.317870    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec"} err="failed to get container status \"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec\": rpc error: code = NotFound desc = could not find container \"853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec\": container with ID starting with 853f39500aa9ff5b98c8ac059c06d28bd227eb983639c41f8464c4ea1c65acec not found: ID does not exist"
Jul 08 16:47:56.318985 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:56.318967    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-77df687bw4gt6"]
Jul 08 16:47:58.245478 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:47:58.245448    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" path="/var/lib/kubelet/pods/9a42218e-b808-4371-a6a0-fe4ce054ca0a/volumes"
Jul 08 16:48:08.742078 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:08.742004    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="main" containerID="cri-o://6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271" gracePeriod=45
Jul 08 16:48:08.860477 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:08.860395    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Jul 08 16:48:09.016397 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.016367    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:48:09.096139 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096153    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096264    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096312    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvt67\" (UniqueName: \"kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67\") pod \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\" (UID: \"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2\") "
Jul 08 16:48:09.096728 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096432    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.096728 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096444    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home" (OuterVolumeSpecName: "home") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.096832 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096736    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.096832 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096756    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.097009 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.096955    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache" (OuterVolumeSpecName: "model-cache") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.098710 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.098685    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm" (OuterVolumeSpecName: "dshm") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.098819 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.098725    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:48:09.098819 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.098747    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67" (OuterVolumeSpecName: "kube-api-access-nvt67") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "kube-api-access-nvt67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:48:09.159751 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.159703    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" (UID: "24863edf-a2eb-4a0a-8453-c8dd91d6dbe2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.197421 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.197380    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.197421 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.197412    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.197421 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.197422    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.197699 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.197431    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvt67\" (UniqueName: \"kubernetes.io/projected/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-kube-api-access-nvt67\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.197699 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.197442    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.351761 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.351722    2559 generic.go:356] "Generic (PLEG): container finished" podID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerID="6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271" exitCode=0
Jul 08 16:48:09.351968 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.351791    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"
Jul 08 16:48:09.351968 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.351803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerDied","Data":"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"}
Jul 08 16:48:09.351968 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.351864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf" event={"ID":"24863edf-a2eb-4a0a-8453-c8dd91d6dbe2","Type":"ContainerDied","Data":"1fe52bcac2f6603e57b45e6d67256b4a6f65460a2bb106e48607e5cc8075288d"}
Jul 08 16:48:09.351968 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.351893    2559 scope.go:117] "RemoveContainer" containerID="6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"
Jul 08 16:48:09.361592 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.361557    2559 scope.go:117] "RemoveContainer" containerID="2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"
Jul 08 16:48:09.370500 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.370482    2559 scope.go:117] "RemoveContainer" containerID="6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"
Jul 08 16:48:09.370828 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:48:09.370808    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271\": container with ID starting with 6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271 not found: ID does not exist" containerID="6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"
Jul 08 16:48:09.370887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.370840    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271"} err="failed to get container status \"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271\": rpc error: code = NotFound desc = could not find container \"6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271\": container with ID starting with 6d1af4668fbccb97e98822e71a98531f88e6e696c6c923c965f02897d394b271 not found: ID does not exist"
Jul 08 16:48:09.370887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.370860    2559 scope.go:117] "RemoveContainer" containerID="2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"
Jul 08 16:48:09.371100 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:48:09.371082    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef\": container with ID starting with 2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef not found: ID does not exist" containerID="2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"
Jul 08 16:48:09.371155 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.371110    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef"} err="failed to get container status \"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef\": rpc error: code = NotFound desc = could not find container \"2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef\": container with ID starting with 2ba92a8a0807127c372ed0bf8ac926f588dfb9c73e326a537838b7eab040deef not found: ID does not exist"
Jul 08 16:48:09.378669 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.378636    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:48:09.380834 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:09.380807    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-68wpf"]
Jul 08 16:48:10.245384 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:10.245350    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" path="/var/lib/kubelet/pods/24863edf-a2eb-4a0a-8453-c8dd91d6dbe2/volumes"
Jul 08 16:48:28.479820 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.479783    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480139    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="storage-initializer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480151    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="storage-initializer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480167    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480183    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480189    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480195    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="tokenizer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480200    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="tokenizer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480208    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480212    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480218    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="storage-initializer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480223    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="storage-initializer"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480231    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d14a7274-03b1-4db8-b906-407b340236e9" containerName="main"
Jul 08 16:48:28.480264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d14a7274-03b1-4db8-b906-407b340236e9" containerName="main"
Jul 08 16:48:28.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480290    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480299    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="tokenizer"
Jul 08 16:48:28.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480306    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d14a7274-03b1-4db8-b906-407b340236e9" containerName="main"
Jul 08 16:48:28.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480311    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a42218e-b808-4371-a6a0-fe4ce054ca0a" containerName="main"
Jul 08 16:48:28.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.480317    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24863edf-a2eb-4a0a-8453-c8dd91d6dbe2" containerName="main"
Jul 08 16:48:28.485041 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.485020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.489394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.489371    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 08 16:48:28.489394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.489386    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:48:28.489596 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.489407    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:48:28.489596 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.489370    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:48:28.493911 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.493878    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:48:28.577064 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.577064 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.577294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24ck7\" (UniqueName: \"kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.577294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.577294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.577294 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.577281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.677910 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.677862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.677910 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.677914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678173 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.677945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678173 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.677964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678173 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.677998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24ck7\" (UniqueName: \"kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678173 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.678048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678382 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.678319    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678427 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.678377    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.678462 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.678441    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.680447 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.680423    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.680739 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.680720    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.688877 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.688851    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24ck7\" (UniqueName: \"kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.795645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.795611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:28.939673 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:28.939636    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:48:28.943120 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:48:28.943089    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd992e4_b3c2_44f3_89fa_6b3c97729926.slice/crio-0fcd51294451c7776e6034cdbd5990025bfba6bd571a5c481d26f8e243af78af WatchSource:0}: Error finding container 0fcd51294451c7776e6034cdbd5990025bfba6bd571a5c481d26f8e243af78af: Status 404 returned error can't find the container with id 0fcd51294451c7776e6034cdbd5990025bfba6bd571a5c481d26f8e243af78af
Jul 08 16:48:29.445028 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:29.444983    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" event={"ID":"7bd992e4-b3c2-44f3-89fa-6b3c97729926","Type":"ContainerStarted","Data":"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"}
Jul 08 16:48:29.445336 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:29.445046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" event={"ID":"7bd992e4-b3c2-44f3-89fa-6b3c97729926","Type":"ContainerStarted","Data":"0fcd51294451c7776e6034cdbd5990025bfba6bd571a5c481d26f8e243af78af"}
Jul 08 16:48:29.465131 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:29.465075    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" podStartSLOduration=1.465057673 podStartE2EDuration="1.465057673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:48:29.463236871 +0000 UTC m=+859.681433391" watchObservedRunningTime="2026-07-08 16:48:29.465057673 +0000 UTC m=+859.683254183"
Jul 08 16:48:29.796018 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:29.795983    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:38.796056 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:38.796020    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:38.809111 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:38.809083    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:48:39.502458 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:48:39.502427    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:49:04.382533 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:04.382441    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:49:10.164689 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:10.164596    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:49:10.169339 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:10.169307    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:49:19.418281 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.418226    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerName="main" containerID="cri-o://e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa" gracePeriod=45
Jul 08 16:49:19.491298 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.491251    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 08 16:49:19.678838 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.678771    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:49:19.683923 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.683893    2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerID="e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa" exitCode=0
Jul 08 16:49:19.684095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.683934    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"
Jul 08 16:49:19.684095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.683954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" event={"ID":"7bd992e4-b3c2-44f3-89fa-6b3c97729926","Type":"ContainerDied","Data":"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"}
Jul 08 16:49:19.684095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.684001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8" event={"ID":"7bd992e4-b3c2-44f3-89fa-6b3c97729926","Type":"ContainerDied","Data":"0fcd51294451c7776e6034cdbd5990025bfba6bd571a5c481d26f8e243af78af"}
Jul 08 16:49:19.684095 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.684033    2559 scope.go:117] "RemoveContainer" containerID="e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"
Jul 08 16:49:19.692736 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.692713    2559 scope.go:117] "RemoveContainer" containerID="e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"
Jul 08 16:49:19.693048 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:49:19.693028    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa\": container with ID starting with e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa not found: ID does not exist" containerID="e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"
Jul 08 16:49:19.693166 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.693059    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa"} err="failed to get container status \"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa\": rpc error: code = NotFound desc = could not find container \"e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa\": container with ID starting with e65193267570304440f9c5193bd5519f36f8fa0faa957a53a755b85b6c65d9aa not found: ID does not exist"
Jul 08 16:49:19.836818 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.836778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24ck7\" (UniqueName: \"kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.836859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.836887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.836939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.836970    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837029 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home\") pod \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\" (UID: \"7bd992e4-b3c2-44f3-89fa-6b3c97729926\") "
Jul 08 16:49:19.837265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837146    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:49:19.837265 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837158    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:49:19.837374 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837337    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home" (OuterVolumeSpecName: "home") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:49:19.837374 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837352    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:19.837374 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.837368    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:19.839282 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.839253    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm" (OuterVolumeSpecName: "dshm") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:49:19.839401 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.839319    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:49:19.839401 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.839339    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7" (OuterVolumeSpecName: "kube-api-access-24ck7") pod "7bd992e4-b3c2-44f3-89fa-6b3c97729926" (UID: "7bd992e4-b3c2-44f3-89fa-6b3c97729926"). InnerVolumeSpecName "kube-api-access-24ck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:49:19.938065 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.937955    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd992e4-b3c2-44f3-89fa-6b3c97729926-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:19.938065 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.937999    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:19.938065 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.938010    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd992e4-b3c2-44f3-89fa-6b3c97729926-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:19.938065 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:19.938019    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24ck7\" (UniqueName: \"kubernetes.io/projected/7bd992e4-b3c2-44f3-89fa-6b3c97729926-kube-api-access-24ck7\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:49:20.010110 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:20.010076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:49:20.017079 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:20.017052    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879cvnz8"]
Jul 08 16:49:20.246110 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:20.246028    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" path="/var/lib/kubelet/pods/7bd992e4-b3c2-44f3-89fa-6b3c97729926/volumes"
Jul 08 16:49:53.985476 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.985441    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:49:53.985953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.985842    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerName="main"
Jul 08 16:49:53.985953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.985856    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerName="main"
Jul 08 16:49:53.985953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.985942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd992e4-b3c2-44f3-89fa-6b3c97729926" containerName="main"
Jul 08 16:49:53.989004 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.988989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:53.992158 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.992138    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:49:53.993334 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.993313    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:49:53.993457 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.993378    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:49:53.993457 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:53.993440    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 08 16:49:54.000390 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.000367    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:49:54.008440 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.008564 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008491    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2b8\" (UniqueName: \"kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.008564 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.008691 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.008691 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.008691 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.008665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.109847 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.109847 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110141 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110141 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110141 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2b8\" (UniqueName: \"kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110141 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.109991    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110353 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.110292    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110394 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.110367    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.110428 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.110391    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.112335 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.112317    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.112617 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.112598    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.120485 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.120459    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2b8\" (UniqueName: \"kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.247977 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.247887    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:49:54.252116 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.252095    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.254826 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.254795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-fsp4n\""
Jul 08 16:49:54.259590 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.259547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:49:54.300076 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.300034    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:49:54.312516 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.312481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp9cw\" (UniqueName: \"kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.312648 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.312619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.312709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.312695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.413360 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.413321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.413551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.413393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.413551 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.413431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp9cw\" (UniqueName: \"kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.413894 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.413860    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.416351 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.416325    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.424385 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.424360    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp9cw\" (UniqueName: \"kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.448241 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.448213    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:49:54.450213 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:49:54.450175    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e198737_43ad_486a_82f8_3684ac34fe90.slice/crio-1be74a705b0b50a9fba1a4095bea13b8ecfd22f1eb8fb3af6cc932f699475196 WatchSource:0}: Error finding container 1be74a705b0b50a9fba1a4095bea13b8ecfd22f1eb8fb3af6cc932f699475196: Status 404 returned error can't find the container with id 1be74a705b0b50a9fba1a4095bea13b8ecfd22f1eb8fb3af6cc932f699475196
Jul 08 16:49:54.571766 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.571727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.715556 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.715481    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:49:54.717915 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:49:54.717870    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4d2fbd_3ac3_4bd2_b836_fdaec6b58543.slice/crio-a23dbb35d075c91bf12b6c2148fc6eac514342e0ad48fe640231aaa7364bd4de WatchSource:0}: Error finding container a23dbb35d075c91bf12b6c2148fc6eac514342e0ad48fe640231aaa7364bd4de: Status 404 returned error can't find the container with id a23dbb35d075c91bf12b6c2148fc6eac514342e0ad48fe640231aaa7364bd4de
Jul 08 16:49:54.857489 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.857453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" event={"ID":"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543","Type":"ContainerStarted","Data":"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"}
Jul 08 16:49:54.857723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.857512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" event={"ID":"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543","Type":"ContainerStarted","Data":"a23dbb35d075c91bf12b6c2148fc6eac514342e0ad48fe640231aaa7364bd4de"}
Jul 08 16:49:54.857723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.857622    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:49:54.859450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.859424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" event={"ID":"6e198737-43ad-486a-82f8-3684ac34fe90","Type":"ContainerStarted","Data":"b40c0c29513fdab2d3367967a2a554455cc8b9642f2d3047fa2bbdfabc58de67"}
Jul 08 16:49:54.859450 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.859472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" event={"ID":"6e198737-43ad-486a-82f8-3684ac34fe90","Type":"ContainerStarted","Data":"1be74a705b0b50a9fba1a4095bea13b8ecfd22f1eb8fb3af6cc932f699475196"}
Jul 08 16:49:54.876545 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.876495    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" podStartSLOduration=0.876478473 podStartE2EDuration="876.478473ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:49:54.875148948 +0000 UTC m=+945.093345458" watchObservedRunningTime="2026-07-08 16:49:54.876478473 +0000 UTC m=+945.094674982"
Jul 08 16:49:54.895927 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:54.895874    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" podStartSLOduration=1.8958564789999999 podStartE2EDuration="1.895856479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:49:54.894864821 +0000 UTC m=+945.113061334" watchObservedRunningTime="2026-07-08 16:49:54.895856479 +0000 UTC m=+945.114052991"
Jul 08 16:49:55.301122 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:49:55.301069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:50:04.300280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:50:04.300242    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:50:04.313147 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:50:04.313120    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:50:04.922130 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:50:04.922099    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:50:25.867864 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:50:25.867831    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:51:52.370925 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.370884    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:51:52.374324 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.374293    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:51:52.383773 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:51:52.383729    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:51:52.383773 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:52.383773 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:52.383773 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8" cmd=["/bin/sleep","15"]
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.383798    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" message="" err=<
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:51:52.383815    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerName="main" containerID="cri-o://e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"
Jul 08 16:51:52.384060 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.383861    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerName="main" containerID="cri-o://e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8" gracePeriod=60
Jul 08 16:51:52.639459 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.639428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:51:52.673751 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.673718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp9cw\" (UniqueName: \"kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw\") pod \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") "
Jul 08 16:51:52.673923 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.673807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds\") pod \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") "
Jul 08 16:51:52.673923 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.673847    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs\") pod \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\" (UID: \"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543\") "
Jul 08 16:51:52.674123 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.674099    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" (UID: "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:51:52.676025 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.675999    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw" (OuterVolumeSpecName: "kube-api-access-pp9cw") pod "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" (UID: "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543"). InnerVolumeSpecName "kube-api-access-pp9cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:51:52.676181 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.676160    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" (UID: "4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:51:52.775144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.775107    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:51:52.775144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.775134    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:51:52.775144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:52.775143    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp9cw\" (UniqueName: \"kubernetes.io/projected/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543-kube-api-access-pp9cw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:51:53.450452 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.450411    2559 generic.go:356] "Generic (PLEG): container finished" podID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerID="e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8" exitCode=0
Jul 08 16:51:53.451007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.450477    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"
Jul 08 16:51:53.451007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.450505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" event={"ID":"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543","Type":"ContainerDied","Data":"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"}
Jul 08 16:51:53.451007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.450560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc" event={"ID":"4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543","Type":"ContainerDied","Data":"a23dbb35d075c91bf12b6c2148fc6eac514342e0ad48fe640231aaa7364bd4de"}
Jul 08 16:51:53.451007 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.450595    2559 scope.go:117] "RemoveContainer" containerID="e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"
Jul 08 16:51:53.460233 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.460212    2559 scope.go:117] "RemoveContainer" containerID="e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"
Jul 08 16:51:53.460543 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:51:53.460526    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8\": container with ID starting with e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8 not found: ID does not exist" containerID="e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"
Jul 08 16:51:53.460637 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.460557    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8"} err="failed to get container status \"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8\": rpc error: code = NotFound desc = could not find container \"e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8\": container with ID starting with e1b88be757dc9437992cdbefc969ef7d53fac8b23495dc58192c0acd1e1763a8 not found: ID does not exist"
Jul 08 16:51:53.476617 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.476567    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:51:53.480706 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:53.480681    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scheckgmc"]
Jul 08 16:51:54.245507 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:51:54.245470    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" path="/var/lib/kubelet/pods/4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543/volumes"
Jul 08 16:52:07.398321 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.398210    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" podUID="6e198737-43ad-486a-82f8-3684ac34fe90" containerName="main" containerID="cri-o://b40c0c29513fdab2d3367967a2a554455cc8b9642f2d3047fa2bbdfabc58de67" gracePeriod=45
Jul 08 16:52:07.525536 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.525505    2559 generic.go:356] "Generic (PLEG): container finished" podID="6e198737-43ad-486a-82f8-3684ac34fe90" containerID="b40c0c29513fdab2d3367967a2a554455cc8b9642f2d3047fa2bbdfabc58de67" exitCode=0
Jul 08 16:52:07.525703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.525603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" event={"ID":"6e198737-43ad-486a-82f8-3684ac34fe90","Type":"ContainerDied","Data":"b40c0c29513fdab2d3367967a2a554455cc8b9642f2d3047fa2bbdfabc58de67"}
Jul 08 16:52:07.658815 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.658738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:52:07.809393 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809353    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809478    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809526    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb2b8\" (UniqueName: \"kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8\") pod \"6e198737-43ad-486a-82f8-3684ac34fe90\" (UID: \"6e198737-43ad-486a-82f8-3684ac34fe90\") "
Jul 08 16:52:07.809914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809723    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:07.809914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809819    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:07.809914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809830    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:07.809914 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.809827    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home" (OuterVolumeSpecName: "home") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:07.811866 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.811837    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm" (OuterVolumeSpecName: "dshm") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:52:07.811866 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.811860    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:52:07.812052 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.811924    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8" (OuterVolumeSpecName: "kube-api-access-cb2b8") pod "6e198737-43ad-486a-82f8-3684ac34fe90" (UID: "6e198737-43ad-486a-82f8-3684ac34fe90"). InnerVolumeSpecName "kube-api-access-cb2b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:52:07.910887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.910785    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:07.910887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.910817    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:07.910887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.910826    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e198737-43ad-486a-82f8-3684ac34fe90-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:07.910887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.910835    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e198737-43ad-486a-82f8-3684ac34fe90-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:07.910887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:07.910844    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2b8\" (UniqueName: \"kubernetes.io/projected/6e198737-43ad-486a-82f8-3684ac34fe90-kube-api-access-cb2b8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:52:08.531498 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:08.531465    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"
Jul 08 16:52:08.531498 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:08.531476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2" event={"ID":"6e198737-43ad-486a-82f8-3684ac34fe90","Type":"ContainerDied","Data":"1be74a705b0b50a9fba1a4095bea13b8ecfd22f1eb8fb3af6cc932f699475196"}
Jul 08 16:52:08.532073 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:08.531543    2559 scope.go:117] "RemoveContainer" containerID="b40c0c29513fdab2d3367967a2a554455cc8b9642f2d3047fa2bbdfabc58de67"
Jul 08 16:52:08.554648 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:08.554616    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:52:08.564382 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:08.564352    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb46pms2"]
Jul 08 16:52:10.244645 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:10.244614    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e198737-43ad-486a-82f8-3684ac34fe90" path="/var/lib/kubelet/pods/6e198737-43ad-486a-82f8-3684ac34fe90/volumes"
Jul 08 16:52:15.588887 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.588852    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:52:15.589260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589201    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e198737-43ad-486a-82f8-3684ac34fe90" containerName="main"
Jul 08 16:52:15.589260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589211    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e198737-43ad-486a-82f8-3684ac34fe90" containerName="main"
Jul 08 16:52:15.589260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589234    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerName="main"
Jul 08 16:52:15.589260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589239    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerName="main"
Jul 08 16:52:15.589408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589291    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4d2fbd-3ac3-4bd2-b836-fdaec6b58543" containerName="main"
Jul 08 16:52:15.589408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.589300    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e198737-43ad-486a-82f8-3684ac34fe90" containerName="main"
Jul 08 16:52:15.593776 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.593760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.597009 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.596986    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:52:15.598283 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.598258    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 08 16:52:15.598418 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.598267    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:52:15.598418 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.598299    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:52:15.606938 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.606915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:52:15.682449 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.682449 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.682709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fm55\" (UniqueName: \"kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.682709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.682709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.682709 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.682544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.783861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.783861 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784135 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fm55\" (UniqueName: \"kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784135 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784135 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784135 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.783983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784388 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.784352    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784388 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.784370    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.784539 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.784403    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.786309 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.786288    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.786715 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.786694    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.796553 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.796528    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fm55\" (UniqueName: \"kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:15.904081 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:15.903983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:16.048953 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.048929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:52:16.050923 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:52:16.050897    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e5886e_9d64_4f20_92b8_b560d0afdce9.slice/crio-cdd04752a1ec98f341e19fc2f9e064cbd9358ec16f2e25c9e8fdd67cb8ce77dc WatchSource:0}: Error finding container cdd04752a1ec98f341e19fc2f9e064cbd9358ec16f2e25c9e8fdd67cb8ce77dc: Status 404 returned error can't find the container with id cdd04752a1ec98f341e19fc2f9e064cbd9358ec16f2e25c9e8fdd67cb8ce77dc
Jul 08 16:52:16.145408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.145370    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:52:16.149287 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.149262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.152014 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.151988    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-mk8bs\""
Jul 08 16:52:16.157924 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.157899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:52:16.288638 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.288601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9sd\" (UniqueName: \"kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.288829 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.288693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.288829 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.288721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.389443 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.389400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9sd\" (UniqueName: \"kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.389636 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.389463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.389636 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.389487    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.389915 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.389887    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.392206 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.392184    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.399690 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.399666    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9sd\" (UniqueName: \"kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.460325 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.460222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:16.573271 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.573238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" event={"ID":"c6e5886e-9d64-4f20-92b8-b560d0afdce9","Type":"ContainerStarted","Data":"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"}
Jul 08 16:52:16.573466 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.573290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" event={"ID":"c6e5886e-9d64-4f20-92b8-b560d0afdce9","Type":"ContainerStarted","Data":"cdd04752a1ec98f341e19fc2f9e064cbd9358ec16f2e25c9e8fdd67cb8ce77dc"}
Jul 08 16:52:16.604952 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.604890    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" podStartSLOduration=1.6048714130000001 podStartE2EDuration="1.604871413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:52:16.597340741 +0000 UTC m=+1086.815537249" watchObservedRunningTime="2026-07-08 16:52:16.604871413 +0000 UTC m=+1086.823067923"
Jul 08 16:52:16.613732 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.613701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:52:16.616243 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:52:16.616198    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646e9504_5d82_4be0_83c0_0b0c675f5929.slice/crio-2e1da4b2dd1dac8cd818284c390677795174d4ca14d6f548b85b6e8a8a7e56d4 WatchSource:0}: Error finding container 2e1da4b2dd1dac8cd818284c390677795174d4ca14d6f548b85b6e8a8a7e56d4: Status 404 returned error can't find the container with id 2e1da4b2dd1dac8cd818284c390677795174d4ca14d6f548b85b6e8a8a7e56d4
Jul 08 16:52:16.904233 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:16.904190    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:17.579166 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:17.579127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" event={"ID":"646e9504-5d82-4be0-83c0-0b0c675f5929","Type":"ContainerStarted","Data":"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"}
Jul 08 16:52:17.579326 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:17.579189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" event={"ID":"646e9504-5d82-4be0-83c0-0b0c675f5929","Type":"ContainerStarted","Data":"2e1da4b2dd1dac8cd818284c390677795174d4ca14d6f548b85b6e8a8a7e56d4"}
Jul 08 16:52:17.579326 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:17.579221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:52:17.601367 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:17.601315    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" podStartSLOduration=1.6012977 podStartE2EDuration="1.6012977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:52:17.599984353 +0000 UTC m=+1087.818180863" watchObservedRunningTime="2026-07-08 16:52:17.6012977 +0000 UTC m=+1087.819494209"
Jul 08 16:52:25.905093 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:25.905054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:25.917731 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:25.917705    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:26.637037 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:26.637008    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:52:48.595644 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:52:48.595608    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:54:10.199559 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:10.199529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:54:10.204604 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:10.204558    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:54:57.313000 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.312957    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:54:57.323945 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.323908    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:54:57.333136 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:54:57.333093    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:54:57.333136 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:54:57.333136 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:54:57.333136 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386" cmd=["/bin/sleep","15"]
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.333153    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" message="" err=<
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:54:57.333169    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerName="main" containerID="cri-o://a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"
Jul 08 16:54:57.333423 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.333218    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerName="main" containerID="cri-o://a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386" gracePeriod=60
Jul 08 16:54:57.603449 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.603424    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:54:57.637107 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.637075    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs\") pod \"646e9504-5d82-4be0-83c0-0b0c675f5929\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") "
Jul 08 16:54:57.637295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.637123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds\") pod \"646e9504-5d82-4be0-83c0-0b0c675f5929\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") "
Jul 08 16:54:57.637295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.637230    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf9sd\" (UniqueName: \"kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd\") pod \"646e9504-5d82-4be0-83c0-0b0c675f5929\" (UID: \"646e9504-5d82-4be0-83c0-0b0c675f5929\") "
Jul 08 16:54:57.637408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.637371    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "646e9504-5d82-4be0-83c0-0b0c675f5929" (UID: "646e9504-5d82-4be0-83c0-0b0c675f5929"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:54:57.637495 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.637482    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/646e9504-5d82-4be0-83c0-0b0c675f5929-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:54:57.639332 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.639309    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd" (OuterVolumeSpecName: "kube-api-access-vf9sd") pod "646e9504-5d82-4be0-83c0-0b0c675f5929" (UID: "646e9504-5d82-4be0-83c0-0b0c675f5929"). InnerVolumeSpecName "kube-api-access-vf9sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:54:57.639408 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.639374    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "646e9504-5d82-4be0-83c0-0b0c675f5929" (UID: "646e9504-5d82-4be0-83c0-0b0c675f5929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:54:57.738941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.738909    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646e9504-5d82-4be0-83c0-0b0c675f5929-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:54:57.738941 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:57.738941    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9sd\" (UniqueName: \"kubernetes.io/projected/646e9504-5d82-4be0-83c0-0b0c675f5929-kube-api-access-vf9sd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:54:58.373019 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.372988    2559 generic.go:356] "Generic (PLEG): container finished" podID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerID="a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386" exitCode=0
Jul 08 16:54:58.373019 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.373023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" event={"ID":"646e9504-5d82-4be0-83c0-0b0c675f5929","Type":"ContainerDied","Data":"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"}
Jul 08 16:54:58.373019 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.373033    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"
Jul 08 16:54:58.373561 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.373060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg" event={"ID":"646e9504-5d82-4be0-83c0-0b0c675f5929","Type":"ContainerDied","Data":"2e1da4b2dd1dac8cd818284c390677795174d4ca14d6f548b85b6e8a8a7e56d4"}
Jul 08 16:54:58.373561 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.373080    2559 scope.go:117] "RemoveContainer" containerID="a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"
Jul 08 16:54:58.382419 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.382400    2559 scope.go:117] "RemoveContainer" containerID="a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"
Jul 08 16:54:58.382708 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:54:58.382687    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386\": container with ID starting with a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386 not found: ID does not exist" containerID="a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"
Jul 08 16:54:58.382756 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.382718    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386"} err="failed to get container status \"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386\": rpc error: code = NotFound desc = could not find container \"a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386\": container with ID starting with a850282aee867480259b6e333d3f22e60856d9600478c31f5d95343b420b7386 not found: ID does not exist"
Jul 08 16:54:58.393628 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.393597    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:54:58.399562 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:54:58.399530    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schep77sg"]
Jul 08 16:55:00.246234 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:00.246200    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" path="/var/lib/kubelet/pods/646e9504-5d82-4be0-83c0-0b0c675f5929/volumes"
Jul 08 16:55:12.339056 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.339014    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" podUID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" containerName="main" containerID="cri-o://631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff" gracePeriod=45
Jul 08 16:55:12.591271 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.591203    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:55:12.669525 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669753 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669555    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fm55\" (UniqueName: \"kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669753 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669596    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669753 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669633    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669753 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669722    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669971 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669760    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs\") pod \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\" (UID: \"c6e5886e-9d64-4f20-92b8-b560d0afdce9\") "
Jul 08 16:55:12.669971 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669889    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home" (OuterVolumeSpecName: "home") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:12.669971 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669895    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:12.669971 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.669954    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:12.670184 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.670119    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:12.670184 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.670135    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:12.670184 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.670150    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:12.671935 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.671910    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm" (OuterVolumeSpecName: "dshm") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:12.672035 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.671973    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55" (OuterVolumeSpecName: "kube-api-access-8fm55") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "kube-api-access-8fm55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:12.672035 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.671990    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6e5886e-9d64-4f20-92b8-b560d0afdce9" (UID: "c6e5886e-9d64-4f20-92b8-b560d0afdce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:12.771260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.771221    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6e5886e-9d64-4f20-92b8-b560d0afdce9-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:12.771260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.771249    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e5886e-9d64-4f20-92b8-b560d0afdce9-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:12.771260 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:12.771260    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fm55\" (UniqueName: \"kubernetes.io/projected/c6e5886e-9d64-4f20-92b8-b560d0afdce9-kube-api-access-8fm55\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:55:13.446921 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.446885    2559 generic.go:356] "Generic (PLEG): container finished" podID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" containerID="631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff" exitCode=0
Jul 08 16:55:13.447416 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.446937    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"
Jul 08 16:55:13.447416 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.446978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" event={"ID":"c6e5886e-9d64-4f20-92b8-b560d0afdce9","Type":"ContainerDied","Data":"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"}
Jul 08 16:55:13.447416 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.447044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b" event={"ID":"c6e5886e-9d64-4f20-92b8-b560d0afdce9","Type":"ContainerDied","Data":"cdd04752a1ec98f341e19fc2f9e064cbd9358ec16f2e25c9e8fdd67cb8ce77dc"}
Jul 08 16:55:13.447416 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.447081    2559 scope.go:117] "RemoveContainer" containerID="631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"
Jul 08 16:55:13.457264 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.457246    2559 scope.go:117] "RemoveContainer" containerID="631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"
Jul 08 16:55:13.457506 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:55:13.457489    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff\": container with ID starting with 631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff not found: ID does not exist" containerID="631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"
Jul 08 16:55:13.457562 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.457514    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff"} err="failed to get container status \"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff\": rpc error: code = NotFound desc = could not find container \"631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff\": container with ID starting with 631cb5f3e27c6cdd3724231d2c9d781ecdf6cf23924a36ad132a8323bc81a6ff not found: ID does not exist"
Jul 08 16:55:13.472801 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.472779    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:55:13.478439 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:13.478416    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-5np5b"]
Jul 08 16:55:14.244652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:14.244616    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" path="/var/lib/kubelet/pods/c6e5886e-9d64-4f20-92b8-b560d0afdce9/volumes"
Jul 08 16:55:27.658295 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658263    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658668    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" containerName="main"
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658684    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" containerName="main"
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658699    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerName="main"
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658707    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerName="main"
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658798    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="646e9504-5d82-4be0-83c0-0b0c675f5929" containerName="main"
Jul 08 16:55:27.658806 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.658808    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e5886e-9d64-4f20-92b8-b560d0afdce9" containerName="main"
Jul 08 16:55:27.663439 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.663419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.667144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.667120    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:55:27.667144 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.667132    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:55:27.667349 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.667212    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-p2kst\""
Jul 08 16:55:27.667349 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.667269    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 08 16:55:27.668650 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.668633    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:55:27.668711 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.668633    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:55:27.676826 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.676804    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:55:27.810852 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.810813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.811040 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.810864    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.811040 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.810930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.811040 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.810989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplmm\" (UniqueName: \"kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912350 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912350 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912308    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912350 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912717 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplmm\" (UniqueName: \"kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912793 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912768    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.912857 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.912823    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.914981 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.914960    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.923005 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.922980    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplmm\" (UniqueName: \"kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:27.972679 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:27.972643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:28.120723 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:28.120699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:55:28.122781 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:55:28.122740    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63c5e7e_b9ce_4165_aae9_4f9f3f70b925.slice/crio-99cadb4cc4e646331451f3d416cf3d9b71934cd41c2d45880a776066444f0a78 WatchSource:0}: Error finding container 99cadb4cc4e646331451f3d416cf3d9b71934cd41c2d45880a776066444f0a78: Status 404 returned error can't find the container with id 99cadb4cc4e646331451f3d416cf3d9b71934cd41c2d45880a776066444f0a78
Jul 08 16:55:28.524522 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:28.524476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerStarted","Data":"c8d08b0b5ccb05900d5441454d35d7ab3b0af100332374b3c73c059549e31576"}
Jul 08 16:55:28.524711 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:28.524531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerStarted","Data":"99cadb4cc4e646331451f3d416cf3d9b71934cd41c2d45880a776066444f0a78"}
Jul 08 16:55:29.530598 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:29.530534    2559 generic.go:356] "Generic (PLEG): container finished" podID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerID="c8d08b0b5ccb05900d5441454d35d7ab3b0af100332374b3c73c059549e31576" exitCode=0
Jul 08 16:55:29.531119 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:29.530627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerDied","Data":"c8d08b0b5ccb05900d5441454d35d7ab3b0af100332374b3c73c059549e31576"}
Jul 08 16:55:30.537356 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:30.537320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerStarted","Data":"0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773"}
Jul 08 16:55:30.537817 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:30.537429    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:55:30.567766 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:55:30.567701    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" podStartSLOduration=2.8692304379999998 podStartE2EDuration="3.567679411s" totalImagesPullingTime="0s" totalInitContainerRuntime="698.448973ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:30.566197147 +0000 UTC m=+1280.784393697" watchObservedRunningTime="2026-07-08 16:55:30.567679411 +0000 UTC m=+1280.785875923"
Jul 08 16:56:01.543594 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:56:01.543540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:57:23.914055 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:23.914017    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:57:23.923775 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:57:23.923738    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:57:23.923775 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:57:23.923775 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:57:23.923775 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773" cmd=["/bin/sleep","15"]
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:23.923788    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" message="" err=<
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:57:23.923799    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="main" containerID="cri-o://0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773"
Jul 08 16:57:23.924048 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:23.923847    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="main" containerID="cri-o://0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773" gracePeriod=60
Jul 08 16:57:24.096024 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.095981    2559 generic.go:356] "Generic (PLEG): container finished" podID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerID="0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773" exitCode=0
Jul 08 16:57:24.096206 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.096049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerDied","Data":"0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773"}
Jul 08 16:57:24.202056 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.202030    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:57:24.361696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.361656    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bplmm\" (UniqueName: \"kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm\") pod \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") "
Jul 08 16:57:24.361696 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.361703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs\") pod \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") "
Jul 08 16:57:24.361976 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.361758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location\") pod \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") "
Jul 08 16:57:24.361976 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.361822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds\") pod \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\" (UID: \"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925\") "
Jul 08 16:57:24.362112 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.362087    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" (UID: "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:24.362474 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.362412    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" (UID: "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:24.364060 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.364036    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm" (OuterVolumeSpecName: "kube-api-access-bplmm") pod "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" (UID: "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925"). InnerVolumeSpecName "kube-api-access-bplmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:24.364156 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.364121    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" (UID: "a63c5e7e-b9ce-4165-aae9-4f9f3f70b925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:24.462919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.462822    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:57:24.462919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.462853    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:57:24.462919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.462863    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplmm\" (UniqueName: \"kubernetes.io/projected/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-kube-api-access-bplmm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:57:24.462919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:24.462873    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:57:25.103170 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.103141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"
Jul 08 16:57:25.103604 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.103140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7" event={"ID":"a63c5e7e-b9ce-4165-aae9-4f9f3f70b925","Type":"ContainerDied","Data":"99cadb4cc4e646331451f3d416cf3d9b71934cd41c2d45880a776066444f0a78"}
Jul 08 16:57:25.103604 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.103270    2559 scope.go:117] "RemoveContainer" containerID="0ae8db3e2745e4acd1b997e250bcb4a61d74fb30205f1b83984057fe79ed6773"
Jul 08 16:57:25.119672 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.119648    2559 scope.go:117] "RemoveContainer" containerID="c8d08b0b5ccb05900d5441454d35d7ab3b0af100332374b3c73c059549e31576"
Jul 08 16:57:25.132652 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.132625    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:57:25.138317 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:25.138293    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schepcpc7"]
Jul 08 16:57:26.244897 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:26.244858    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" path="/var/lib/kubelet/pods/a63c5e7e-b9ce-4165-aae9-4f9f3f70b925/volumes"
Jul 08 16:57:57.138522 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138473    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:57:57.139157 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138875    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="main"
Jul 08 16:57:57.139157 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138888    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="main"
Jul 08 16:57:57.139157 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138916    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="storage-initializer"
Jul 08 16:57:57.139157 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138926    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="storage-initializer"
Jul 08 16:57:57.139157 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.138983    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a63c5e7e-b9ce-4165-aae9-4f9f3f70b925" containerName="main"
Jul 08 16:57:57.142212 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.142187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.145167 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.145141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:57:57.145438 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.145423    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:57:57.145539 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.145439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:57:57.146618 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.146567    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:57:57.146618 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.146567    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 08 16:57:57.146809 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.146598    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-c85qg\""
Jul 08 16:57:57.154370 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.154350    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:57:57.254472 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.254433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.254472 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.254485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.254801 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.254647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.254801 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.254684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6fd\" (UniqueName: \"kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.355703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.355666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.355703 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.355705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6fd\" (UniqueName: \"kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.355976 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.355732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.355976 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.355867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.356114 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.356093    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.356160 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.356119    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.358432 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.358413    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.365181 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.365158    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6fd\" (UniqueName: \"kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.452974 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.452865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:57:57.596268 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:57.596240    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:57:57.598817 ip-10-0-130-182 kubenswrapper[2559]: W0708 16:57:57.598789    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635102fd_c3ff_4f72_afe7_36e2f1670864.slice/crio-408ebd77e28be539c32f30f4a0cc7a2511e19d44359a4f0c2415d537c109e57a WatchSource:0}: Error finding container 408ebd77e28be539c32f30f4a0cc7a2511e19d44359a4f0c2415d537c109e57a: Status 404 returned error can't find the container with id 408ebd77e28be539c32f30f4a0cc7a2511e19d44359a4f0c2415d537c109e57a
Jul 08 16:57:58.263006 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:58.262964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerStarted","Data":"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"}
Jul 08 16:57:58.263410 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:58.263023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerStarted","Data":"408ebd77e28be539c32f30f4a0cc7a2511e19d44359a4f0c2415d537c109e57a"}
Jul 08 16:57:59.269317 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:59.269280    2559 generic.go:356] "Generic (PLEG): container finished" podID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerID="75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8" exitCode=0
Jul 08 16:57:59.269759 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:57:59.269346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerDied","Data":"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"}
Jul 08 16:58:00.276433 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:58:00.276397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerStarted","Data":"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"}
Jul 08 16:58:00.276888 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:58:00.276521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:58:00.307315 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:58:00.307265    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" podStartSLOduration=2.604235187 podStartE2EDuration="3.307248027s" totalImagesPullingTime="0s" totalInitContainerRuntime="703.01284ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:00.305171106 +0000 UTC m=+1430.523367616" watchObservedRunningTime="2026-07-08 16:58:00.307248027 +0000 UTC m=+1430.525444536"
Jul 08 16:58:31.283319 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:58:31.283291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:59:10.229841 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:10.229808    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:59:10.236364 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:10.236325    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 16:59:38.889612 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:38.889513    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:59:38.898941 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:59:38.898899    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:59:38.898941 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:59:38.898941 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:59:38.898941 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe" cmd=["/bin/sleep","15"]
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:38.898952    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" message="" err=<
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:59:38.898966    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="main" containerID="cri-o://e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"
Jul 08 16:59:38.899235 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:38.899013    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="main" containerID="cri-o://e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe" gracePeriod=60
Jul 08 16:59:39.164300 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.164275    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:59:39.273332 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.273300    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs\") pod \"635102fd-c3ff-4f72-afe7-36e2f1670864\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") "
Jul 08 16:59:39.273332 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.273347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds\") pod \"635102fd-c3ff-4f72-afe7-36e2f1670864\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") "
Jul 08 16:59:39.273653 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.273389    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf6fd\" (UniqueName: \"kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd\") pod \"635102fd-c3ff-4f72-afe7-36e2f1670864\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") "
Jul 08 16:59:39.273653 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.273418    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location\") pod \"635102fd-c3ff-4f72-afe7-36e2f1670864\" (UID: \"635102fd-c3ff-4f72-afe7-36e2f1670864\") "
Jul 08 16:59:39.273779 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.273695    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "635102fd-c3ff-4f72-afe7-36e2f1670864" (UID: "635102fd-c3ff-4f72-afe7-36e2f1670864"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:39.274223 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.274198    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "635102fd-c3ff-4f72-afe7-36e2f1670864" (UID: "635102fd-c3ff-4f72-afe7-36e2f1670864"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:39.275808 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.275776    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd" (OuterVolumeSpecName: "kube-api-access-tf6fd") pod "635102fd-c3ff-4f72-afe7-36e2f1670864" (UID: "635102fd-c3ff-4f72-afe7-36e2f1670864"). InnerVolumeSpecName "kube-api-access-tf6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:39.275919 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.275824    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "635102fd-c3ff-4f72-afe7-36e2f1670864" (UID: "635102fd-c3ff-4f72-afe7-36e2f1670864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:39.374180 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.374140    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/635102fd-c3ff-4f72-afe7-36e2f1670864-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:59:39.374389 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.374277    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:59:39.374452 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.374411    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf6fd\" (UniqueName: \"kubernetes.io/projected/635102fd-c3ff-4f72-afe7-36e2f1670864-kube-api-access-tf6fd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:59:39.374452 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.374432    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/635102fd-c3ff-4f72-afe7-36e2f1670864-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 16:59:39.789280 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.789237    2559 generic.go:356] "Generic (PLEG): container finished" podID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerID="e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe" exitCode=0
Jul 08 16:59:39.789475 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.789282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerDied","Data":"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"}
Jul 08 16:59:39.789475 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.789311    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"
Jul 08 16:59:39.789475 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.789361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt" event={"ID":"635102fd-c3ff-4f72-afe7-36e2f1670864","Type":"ContainerDied","Data":"408ebd77e28be539c32f30f4a0cc7a2511e19d44359a4f0c2415d537c109e57a"}
Jul 08 16:59:39.789475 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.789391    2559 scope.go:117] "RemoveContainer" containerID="e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"
Jul 08 16:59:39.799152 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.799134    2559 scope.go:117] "RemoveContainer" containerID="75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"
Jul 08 16:59:39.807071 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.807052    2559 scope.go:117] "RemoveContainer" containerID="e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"
Jul 08 16:59:39.807361 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:59:39.807341    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe\": container with ID starting with e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe not found: ID does not exist" containerID="e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"
Jul 08 16:59:39.807419 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.807376    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe"} err="failed to get container status \"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe\": rpc error: code = NotFound desc = could not find container \"e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe\": container with ID starting with e403c4da704dab62fc197ced18298b4a1286c7de43f9c82db5fb763285244cfe not found: ID does not exist"
Jul 08 16:59:39.807419 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.807395    2559 scope.go:117] "RemoveContainer" containerID="75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"
Jul 08 16:59:39.807748 ip-10-0-130-182 kubenswrapper[2559]: E0708 16:59:39.807728    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8\": container with ID starting with 75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8 not found: ID does not exist" containerID="75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"
Jul 08 16:59:39.807815 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.807756    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8"} err="failed to get container status \"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8\": rpc error: code = NotFound desc = could not find container \"75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8\": container with ID starting with 75ef207f4671995acf928c750f9f5eb1ddb98b2fe14f4522f681efa39b929ad8 not found: ID does not exist"
Jul 08 16:59:39.815382 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.815359    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:59:39.823239 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:39.823219    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schen2sbt"]
Jul 08 16:59:40.244347 ip-10-0-130-182 kubenswrapper[2559]: I0708 16:59:40.244263    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" path="/var/lib/kubelet/pods/635102fd-c3ff-4f72-afe7-36e2f1670864/volumes"
Jul 08 17:00:02.337984 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.337948    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"]
Jul 08 17:00:02.338429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.338307    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="storage-initializer"
Jul 08 17:00:02.338429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.338318    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="storage-initializer"
Jul 08 17:00:02.338429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.338328    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="main"
Jul 08 17:00:02.338429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.338333    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="main"
Jul 08 17:00:02.338429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.338387    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="635102fd-c3ff-4f72-afe7-36e2f1670864" containerName="main"
Jul 08 17:00:02.342972 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.342946    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:02.346098 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.346075    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:00:02.347258 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.347233    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:00:02.347371 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.347233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:00:02.349861 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.349812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"]
Jul 08 17:00:02.377638 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.377598    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245z9\" (UniqueName: \"kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:02.478845 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.478808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245z9\" (UniqueName: \"kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:02.494429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:02.494400    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245z9\" (UniqueName: \"kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.436282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.436236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.447474 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.447444    2559 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.537282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.537241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.537497 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.537383    2559 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.542113 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.542087    2559 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") DevicePath \"csi-59a755afb9669f25ef7a6a36bae4ad52254a25fda3fb521b28038f769a546729\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.774275 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.774237    2559 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4c2afd44bc1329e2e7018951f0ca78b5032d86e3fdf6923f6d1397e5ac2aa83/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:10.792644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:10.792615    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e2e-pvc-model-download-bsd5v\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:11.070991 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.070902    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:00:11.076418 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.076398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:11.224842 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.224809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"]
Jul 08 17:00:11.228236 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:00:11.228208    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod476696a4_2052_4220_80c5_0524c8298818.slice/crio-f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500 WatchSource:0}: Error finding container f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500: Status 404 returned error can't find the container with id f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500
Jul 08 17:00:11.952197 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.952160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v" event={"ID":"476696a4-2052-4220-80c5-0524c8298818","Type":"ContainerStarted","Data":"33aadd556f86cc32d41bb71fb91a30a05ebe55412c5d7f3ea6b840c8374aca3c"}
Jul 08 17:00:11.952631 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.952206    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v" event={"ID":"476696a4-2052-4220-80c5-0524c8298818","Type":"ContainerStarted","Data":"f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500"}
Jul 08 17:00:11.980908 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:11.980846    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v" podStartSLOduration=12.980825246 podStartE2EDuration="12.980825246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:00:11.98050206 +0000 UTC m=+1562.198698568" watchObservedRunningTime="2026-07-08 17:00:11.980825246 +0000 UTC m=+1562.199021756"
Jul 08 17:00:17.983210 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:17.983117    2559 generic.go:356] "Generic (PLEG): container finished" podID="476696a4-2052-4220-80c5-0524c8298818" containerID="33aadd556f86cc32d41bb71fb91a30a05ebe55412c5d7f3ea6b840c8374aca3c" exitCode=0
Jul 08 17:00:17.983210 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:17.983192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v" event={"ID":"476696a4-2052-4220-80c5-0524c8298818","Type":"ContainerDied","Data":"33aadd556f86cc32d41bb71fb91a30a05ebe55412c5d7f3ea6b840c8374aca3c"}
Jul 08 17:00:19.125847 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.125824    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:19.317787 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.317763    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"476696a4-2052-4220-80c5-0524c8298818\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") "
Jul 08 17:00:19.317965 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.317821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-245z9\" (UniqueName: \"kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9\") pod \"476696a4-2052-4220-80c5-0524c8298818\" (UID: \"476696a4-2052-4220-80c5-0524c8298818\") "
Jul 08 17:00:19.320355 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.320305    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9" (OuterVolumeSpecName: "kube-api-access-245z9") pod "476696a4-2052-4220-80c5-0524c8298818" (UID: "476696a4-2052-4220-80c5-0524c8298818"). InnerVolumeSpecName "kube-api-access-245z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:00:19.321835 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.321816    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "model-storage") pod "476696a4-2052-4220-80c5-0524c8298818" (UID: "476696a4-2052-4220-80c5-0524c8298818"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:00:19.419821 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.419783    2559 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-130-182.ec2.internal\" "
Jul 08 17:00:19.419821 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.419824    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245z9\" (UniqueName: \"kubernetes.io/projected/476696a4-2052-4220-80c5-0524c8298818-kube-api-access-245z9\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:00:19.530558 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.530536    2559 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75") on node "ip-10-0-130-182.ec2.internal"
Jul 08 17:00:19.621449 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.621365    2559 reconciler_common.go:299] "Volume detached for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"csi-59a755afb9669f25ef7a6a36bae4ad52254a25fda3fb521b28038f769a546729\""
Jul 08 17:00:19.996378 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.996257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v" event={"ID":"476696a4-2052-4220-80c5-0524c8298818","Type":"ContainerDied","Data":"f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500"}
Jul 08 17:00:19.996378 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.996311    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7efaa395879fadd9dd34b9362c50b57d87987504561232f25bd252d27f04500"
Jul 08 17:00:19.996708 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:19.996410    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-bsd5v"
Jul 08 17:00:40.103412 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.103373    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:00:40.103997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.103835    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476696a4-2052-4220-80c5-0524c8298818" containerName="storage-initializer"
Jul 08 17:00:40.103997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.103854    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="476696a4-2052-4220-80c5-0524c8298818" containerName="storage-initializer"
Jul 08 17:00:40.103997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.103945    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="476696a4-2052-4220-80c5-0524c8298818" containerName="storage-initializer"
Jul 08 17:00:40.107411 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.107388    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.111815 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.111782    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:00:40.111815 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.111800    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:00:40.111815 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.111805    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:00:40.112251 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.111801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 08 17:00:40.112399 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.112341    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-fbtqb\""
Jul 08 17:00:40.122934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.122911    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:00:40.194918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.194882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kql2x\" (UniqueName: \"kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.195117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.194941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.195117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.194981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.295478 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.295446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kql2x\" (UniqueName: \"kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.295711 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.295511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.295711 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.295533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.296019 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.296000    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.298257 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.298229    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.308971 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.308947    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kql2x\" (UniqueName: \"kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.425121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.425021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:40.577818 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:40.577700    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:00:40.581043 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:00:40.581002    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c83e42_ba0d_4a56_baa3_707973fc5396.slice/crio-dee6663a6fc2eab6e9366ee89a30849fca5ce315db03e63f429b2edd5f40e64a WatchSource:0}: Error finding container dee6663a6fc2eab6e9366ee89a30849fca5ce315db03e63f429b2edd5f40e64a: Status 404 returned error can't find the container with id dee6663a6fc2eab6e9366ee89a30849fca5ce315db03e63f429b2edd5f40e64a
Jul 08 17:00:41.099385 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:41.099342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" event={"ID":"a5c83e42-ba0d-4a56-baa3-707973fc5396","Type":"ContainerStarted","Data":"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"}
Jul 08 17:00:41.099601 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:41.099410    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:00:41.099601 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:41.099430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" event={"ID":"a5c83e42-ba0d-4a56-baa3-707973fc5396","Type":"ContainerStarted","Data":"dee6663a6fc2eab6e9366ee89a30849fca5ce315db03e63f429b2edd5f40e64a"}
Jul 08 17:00:41.130241 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:00:41.130179    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" podStartSLOduration=1.130163406 podStartE2EDuration="1.130163406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:00:41.125498224 +0000 UTC m=+1591.343694733" watchObservedRunningTime="2026-07-08 17:00:41.130163406 +0000 UTC m=+1591.348359914"
Jul 08 17:01:12.105769 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:01:12.105694    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:02:41.419413 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.419327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:02:41.428668 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:02:41.428627    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:02:41.428668 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:02:41.428668 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:02:41.428668 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b" cmd=["/bin/sleep","15"]
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.428693    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" message="" err=<
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:02:41.428710    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerName="main" containerID="cri-o://a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"
Jul 08 17:02:41.428936 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.428785    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerName="main" containerID="cri-o://a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b" gracePeriod=60
Jul 08 17:02:41.679230 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.679166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:02:41.687743 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.687717    2559 generic.go:356] "Generic (PLEG): container finished" podID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerID="a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b" exitCode=0
Jul 08 17:02:41.687899 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.687764    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"
Jul 08 17:02:41.687899 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.687795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" event={"ID":"a5c83e42-ba0d-4a56-baa3-707973fc5396","Type":"ContainerDied","Data":"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"}
Jul 08 17:02:41.687899 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.687848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7" event={"ID":"a5c83e42-ba0d-4a56-baa3-707973fc5396","Type":"ContainerDied","Data":"dee6663a6fc2eab6e9366ee89a30849fca5ce315db03e63f429b2edd5f40e64a"}
Jul 08 17:02:41.687899 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.687871    2559 scope.go:117] "RemoveContainer" containerID="a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"
Jul 08 17:02:41.696405 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.696388    2559 scope.go:117] "RemoveContainer" containerID="a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"
Jul 08 17:02:41.696773 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:02:41.696755    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b\": container with ID starting with a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b not found: ID does not exist" containerID="a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"
Jul 08 17:02:41.696848 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.696785    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b"} err="failed to get container status \"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b\": rpc error: code = NotFound desc = could not find container \"a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b\": container with ID starting with a27ffd40ad5c9299eb63a9dd7ee310f2ba514073b2f4dff21e3be3d7b477a10b not found: ID does not exist"
Jul 08 17:02:41.766427 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.766392    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs\") pod \"a5c83e42-ba0d-4a56-baa3-707973fc5396\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") "
Jul 08 17:02:41.766630 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.766442    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kql2x\" (UniqueName: \"kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x\") pod \"a5c83e42-ba0d-4a56-baa3-707973fc5396\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") "
Jul 08 17:02:41.766630 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.766475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds\") pod \"a5c83e42-ba0d-4a56-baa3-707973fc5396\" (UID: \"a5c83e42-ba0d-4a56-baa3-707973fc5396\") "
Jul 08 17:02:41.766844 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.766819    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a5c83e42-ba0d-4a56-baa3-707973fc5396" (UID: "a5c83e42-ba0d-4a56-baa3-707973fc5396"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:02:41.768727 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.768695    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5c83e42-ba0d-4a56-baa3-707973fc5396" (UID: "a5c83e42-ba0d-4a56-baa3-707973fc5396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:02:41.768815 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.768749    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x" (OuterVolumeSpecName: "kube-api-access-kql2x") pod "a5c83e42-ba0d-4a56-baa3-707973fc5396" (UID: "a5c83e42-ba0d-4a56-baa3-707973fc5396"). InnerVolumeSpecName "kube-api-access-kql2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:02:41.867649 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.867613    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a5c83e42-ba0d-4a56-baa3-707973fc5396-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:02:41.867649 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.867648    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c83e42-ba0d-4a56-baa3-707973fc5396-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:02:41.867949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:41.867662    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kql2x\" (UniqueName: \"kubernetes.io/projected/a5c83e42-ba0d-4a56-baa3-707973fc5396-kube-api-access-kql2x\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:02:42.017125 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:42.017089    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:02:42.021226 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:42.021201    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schehqcg7"]
Jul 08 17:02:42.251615 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:42.251512    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" path="/var/lib/kubelet/pods/a5c83e42-ba0d-4a56-baa3-707973fc5396/volumes"
Jul 08 17:02:53.536422 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.536387    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:02:53.536822 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.536766    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerName="main"
Jul 08 17:02:53.536822 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.536778    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerName="main"
Jul 08 17:02:53.536891 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.536847    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c83e42-ba0d-4a56-baa3-707973fc5396" containerName="main"
Jul 08 17:02:53.541272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.541251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.544193 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.544171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-8bx58\""
Jul 08 17:02:53.545252 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.545236    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:02:53.545368 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.545335    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:02:53.545368 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.545334    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 08 17:02:53.545545 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.545338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:02:53.545545 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.545339    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:02:53.555253 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.555225    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:02:53.680254 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.680215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dk6q\" (UniqueName: \"kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.680434 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.680260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.680434 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.680341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.680434 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.680389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.781797 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.781761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dk6q\" (UniqueName: \"kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.781797 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.781800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.782057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.781843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.782057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.781900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.782259 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.782240    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.782319 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.782299    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.784497 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.784481    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.794276 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.794208    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dk6q\" (UniqueName: \"kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:53.851445 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:53.851410    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:54.002928 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:54.002906    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:02:54.005963 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:02:54.005906    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f15f44_a3c0_477a_9c85_dff4f47f12bb.slice/crio-064e99ca7024ed1eb864bfffe1c531251584a08bb10e35211d3b3f294337d5a2 WatchSource:0}: Error finding container 064e99ca7024ed1eb864bfffe1c531251584a08bb10e35211d3b3f294337d5a2: Status 404 returned error can't find the container with id 064e99ca7024ed1eb864bfffe1c531251584a08bb10e35211d3b3f294337d5a2
Jul 08 17:02:54.754179 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:54.754143    2559 generic.go:356] "Generic (PLEG): container finished" podID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerID="76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da" exitCode=0
Jul 08 17:02:54.754635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:54.754230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" event={"ID":"19f15f44-a3c0-477a-9c85-dff4f47f12bb","Type":"ContainerDied","Data":"76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da"}
Jul 08 17:02:54.754635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:54.754306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" event={"ID":"19f15f44-a3c0-477a-9c85-dff4f47f12bb","Type":"ContainerStarted","Data":"064e99ca7024ed1eb864bfffe1c531251584a08bb10e35211d3b3f294337d5a2"}
Jul 08 17:02:55.761062 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:55.761027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" event={"ID":"19f15f44-a3c0-477a-9c85-dff4f47f12bb","Type":"ContainerStarted","Data":"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"}
Jul 08 17:02:55.761463 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:55.761147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:02:55.802419 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:02:55.802363    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" podStartSLOduration=2.138145904 podStartE2EDuration="2.802346987s" totalImagesPullingTime="0s" totalInitContainerRuntime="664.201083ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:02:55.801388102 +0000 UTC m=+1726.019584649" watchObservedRunningTime="2026-07-08 17:02:55.802346987 +0000 UTC m=+1726.020543507"
Jul 08 17:03:26.767972 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:03:26.767939    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:04:10.260765 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:04:10.260733    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:04:10.269035 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:04:10.269010    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:05:44.757182 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:44.757093    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:05:44.767428 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:05:44.767391    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:05:44.767428 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:05:44.767428 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:05:44.767428 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f" cmd=["/bin/sleep","15"]
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:44.767443    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" message="" err=<
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:05:44.767455    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="main" containerID="cri-o://feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"
Jul 08 17:05:44.767783 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:44.767508    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="main" containerID="cri-o://feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f" gracePeriod=60
Jul 08 17:05:45.026358 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.026329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:05:45.109083 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds\") pod \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") "
Jul 08 17:05:45.109282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109119    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs\") pod \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") "
Jul 08 17:05:45.109282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109160    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dk6q\" (UniqueName: \"kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q\") pod \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") "
Jul 08 17:05:45.109282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109221    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location\") pod \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\" (UID: \"19f15f44-a3c0-477a-9c85-dff4f47f12bb\") "
Jul 08 17:05:45.109451 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109348    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "19f15f44-a3c0-477a-9c85-dff4f47f12bb" (UID: "19f15f44-a3c0-477a-9c85-dff4f47f12bb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:05:45.109636 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.109608    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:05:45.110139 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.110113    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19f15f44-a3c0-477a-9c85-dff4f47f12bb" (UID: "19f15f44-a3c0-477a-9c85-dff4f47f12bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:05:45.112274 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.112242    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19f15f44-a3c0-477a-9c85-dff4f47f12bb" (UID: "19f15f44-a3c0-477a-9c85-dff4f47f12bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:05:45.112385 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.112278    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q" (OuterVolumeSpecName: "kube-api-access-8dk6q") pod "19f15f44-a3c0-477a-9c85-dff4f47f12bb" (UID: "19f15f44-a3c0-477a-9c85-dff4f47f12bb"). InnerVolumeSpecName "kube-api-access-8dk6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:05:45.210795 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.210759    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f15f44-a3c0-477a-9c85-dff4f47f12bb-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:05:45.210795 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.210790    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dk6q\" (UniqueName: \"kubernetes.io/projected/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kube-api-access-8dk6q\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:05:45.210795 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.210801    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19f15f44-a3c0-477a-9c85-dff4f47f12bb-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:05:45.586548 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.586514    2559 generic.go:356] "Generic (PLEG): container finished" podID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerID="feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f" exitCode=0
Jul 08 17:05:45.586752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.586581    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"
Jul 08 17:05:45.586752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.586610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" event={"ID":"19f15f44-a3c0-477a-9c85-dff4f47f12bb","Type":"ContainerDied","Data":"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"}
Jul 08 17:05:45.586752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.586665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2" event={"ID":"19f15f44-a3c0-477a-9c85-dff4f47f12bb","Type":"ContainerDied","Data":"064e99ca7024ed1eb864bfffe1c531251584a08bb10e35211d3b3f294337d5a2"}
Jul 08 17:05:45.586752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.586690    2559 scope.go:117] "RemoveContainer" containerID="feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"
Jul 08 17:05:45.598991 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.598968    2559 scope.go:117] "RemoveContainer" containerID="76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da"
Jul 08 17:05:45.608955 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.608935    2559 scope.go:117] "RemoveContainer" containerID="feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"
Jul 08 17:05:45.609272 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:05:45.609252    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f\": container with ID starting with feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f not found: ID does not exist" containerID="feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"
Jul 08 17:05:45.609350 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.609286    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f"} err="failed to get container status \"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f\": rpc error: code = NotFound desc = could not find container \"feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f\": container with ID starting with feb2647cb1fc2409b365d70c3e0e69e884c21d00c7fdbfd95ff4e9f2b1fe7e3f not found: ID does not exist"
Jul 08 17:05:45.609350 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.609316    2559 scope.go:117] "RemoveContainer" containerID="76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da"
Jul 08 17:05:45.609609 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:05:45.609564    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da\": container with ID starting with 76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da not found: ID does not exist" containerID="76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da"
Jul 08 17:05:45.609707 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.609615    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da"} err="failed to get container status \"76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da\": rpc error: code = NotFound desc = could not find container \"76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da\": container with ID starting with 76e1bfb27efc23d329c5cbb3074c6ae69d47ec76cab8ae7e188ea199585634da not found: ID does not exist"
Jul 08 17:05:45.612792 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.612769    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:05:45.617545 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:45.617521    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-sche58jw2"]
Jul 08 17:05:46.245136 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:05:46.245102    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" path="/var/lib/kubelet/pods/19f15f44-a3c0-477a-9c85-dff4f47f12bb/volumes"
Jul 08 17:06:16.267107 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267073    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:06:16.267686 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267612    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="storage-initializer"
Jul 08 17:06:16.267686 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267632    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="storage-initializer"
Jul 08 17:06:16.267686 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="main"
Jul 08 17:06:16.267686 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267670    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="main"
Jul 08 17:06:16.267917 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.267796    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19f15f44-a3c0-477a-9c85-dff4f47f12bb" containerName="main"
Jul 08 17:06:16.272274 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.272249    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.276741    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.276936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.277051    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.277255    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-pr9rs\""
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.277782    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 08 17:06:16.281860 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.277971    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:06:16.286123 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.286091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:06:16.391459 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.391422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.391662 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.391555    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.391662 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.391629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.391662 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.391652    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjfn\" (UniqueName: \"kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493009 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.492955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493009 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.493012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.493031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjfn\" (UniqueName: \"kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.493063    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493383 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.493359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.493428 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.493412    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.495682 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.495660    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.504053 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.504024    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjfn\" (UniqueName: \"kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn\") pod \"custom-route-timeout-test-kserve-router-scheduler-664966fctvzts\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.590317 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.590277    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:16.740834 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:16.740812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:06:16.742771 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:06:16.742738    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13102fb4_9acb_4c99_bde3_d53167ba0f6f.slice/crio-e898856306c86f8d71105ee801de56a0690aaeefc55b9b6b37b765b79fd330c9 WatchSource:0}: Error finding container e898856306c86f8d71105ee801de56a0690aaeefc55b9b6b37b765b79fd330c9: Status 404 returned error can't find the container with id e898856306c86f8d71105ee801de56a0690aaeefc55b9b6b37b765b79fd330c9
Jul 08 17:06:17.742465 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:17.742426    2559 generic.go:356] "Generic (PLEG): container finished" podID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerID="ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293" exitCode=0
Jul 08 17:06:17.742906 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:17.742511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" event={"ID":"13102fb4-9acb-4c99-bde3-d53167ba0f6f","Type":"ContainerDied","Data":"ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293"}
Jul 08 17:06:17.742906 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:17.742589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" event={"ID":"13102fb4-9acb-4c99-bde3-d53167ba0f6f","Type":"ContainerStarted","Data":"e898856306c86f8d71105ee801de56a0690aaeefc55b9b6b37b765b79fd330c9"}
Jul 08 17:06:18.749377 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:18.749344    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" event={"ID":"13102fb4-9acb-4c99-bde3-d53167ba0f6f","Type":"ContainerStarted","Data":"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"}
Jul 08 17:06:18.749882 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:18.749418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:06:18.774361 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:18.774298    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" podStartSLOduration=2.072253651 podStartE2EDuration="2.77428071s" totalImagesPullingTime="0s" totalInitContainerRuntime="702.027059ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:06:18.772458812 +0000 UTC m=+1928.990655360" watchObservedRunningTime="2026-07-08 17:06:18.77428071 +0000 UTC m=+1928.992477218"
Jul 08 17:06:49.755417 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:06:49.755326    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:09:09.078816 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.078781    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:09:09.088603 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:09:09.088537    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:09:09.088603 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:09:09.088603 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:09:09.088603 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c" cmd=["/bin/sleep","15"]
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.088624    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" message="" err=<
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:09:09.088643    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="main" containerID="cri-o://cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"
Jul 08 17:09:09.088892 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.088696    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="main" containerID="cri-o://cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c" gracePeriod=60
Jul 08 17:09:09.349070 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.349041    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:09:09.431329 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.431290    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location\") pod \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") "
Jul 08 17:09:09.431329 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.431335    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs\") pod \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") "
Jul 08 17:09:09.431631 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.431378    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnjfn\" (UniqueName: \"kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn\") pod \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") "
Jul 08 17:09:09.431631 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.431428    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds\") pod \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\" (UID: \"13102fb4-9acb-4c99-bde3-d53167ba0f6f\") "
Jul 08 17:09:09.431817 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.431789    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "13102fb4-9acb-4c99-bde3-d53167ba0f6f" (UID: "13102fb4-9acb-4c99-bde3-d53167ba0f6f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:09.432091 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.432066    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13102fb4-9acb-4c99-bde3-d53167ba0f6f" (UID: "13102fb4-9acb-4c99-bde3-d53167ba0f6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:09.433941 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.433914    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13102fb4-9acb-4c99-bde3-d53167ba0f6f" (UID: "13102fb4-9acb-4c99-bde3-d53167ba0f6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:09:09.434037 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.434012    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn" (OuterVolumeSpecName: "kube-api-access-nnjfn") pod "13102fb4-9acb-4c99-bde3-d53167ba0f6f" (UID: "13102fb4-9acb-4c99-bde3-d53167ba0f6f"). InnerVolumeSpecName "kube-api-access-nnjfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:09:09.532677 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.532643    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:09.532677 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.532673    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:09.532677 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.532683    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13102fb4-9acb-4c99-bde3-d53167ba0f6f-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:09.532937 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.532694    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjfn\" (UniqueName: \"kubernetes.io/projected/13102fb4-9acb-4c99-bde3-d53167ba0f6f-kube-api-access-nnjfn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:09.589098 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.588998    2559 generic.go:356] "Generic (PLEG): container finished" podID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerID="cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c" exitCode=0
Jul 08 17:09:09.589098 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.589055    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"
Jul 08 17:09:09.589320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.589088    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" event={"ID":"13102fb4-9acb-4c99-bde3-d53167ba0f6f","Type":"ContainerDied","Data":"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"}
Jul 08 17:09:09.589320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.589147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts" event={"ID":"13102fb4-9acb-4c99-bde3-d53167ba0f6f","Type":"ContainerDied","Data":"e898856306c86f8d71105ee801de56a0690aaeefc55b9b6b37b765b79fd330c9"}
Jul 08 17:09:09.589320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.589168    2559 scope.go:117] "RemoveContainer" containerID="cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"
Jul 08 17:09:09.599001 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.598983    2559 scope.go:117] "RemoveContainer" containerID="ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293"
Jul 08 17:09:09.609027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.609004    2559 scope.go:117] "RemoveContainer" containerID="cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"
Jul 08 17:09:09.609365 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:09:09.609345    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c\": container with ID starting with cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c not found: ID does not exist" containerID="cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"
Jul 08 17:09:09.609417 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.609375    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c"} err="failed to get container status \"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c\": rpc error: code = NotFound desc = could not find container \"cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c\": container with ID starting with cf9c62dd5aaf747a8608f7113080251d5a8baa0cdfc49ed93c0156380605d87c not found: ID does not exist"
Jul 08 17:09:09.609417 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.609394    2559 scope.go:117] "RemoveContainer" containerID="ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293"
Jul 08 17:09:09.609666 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:09:09.609648    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293\": container with ID starting with ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293 not found: ID does not exist" containerID="ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293"
Jul 08 17:09:09.609717 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.609672    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293"} err="failed to get container status \"ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293\": rpc error: code = NotFound desc = could not find container \"ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293\": container with ID starting with ac0b0b91a99e66229d530712c8cce30602534f46e14cd7bebc0d537788a20293 not found: ID does not exist"
Jul 08 17:09:09.613597 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.613557    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:09:09.618668 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:09.618643    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-664966fctvzts"]
Jul 08 17:09:10.245627 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:10.245590    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" path="/var/lib/kubelet/pods/13102fb4-9acb-4c99-bde3-d53167ba0f6f/volumes"
Jul 08 17:09:10.290587 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:10.290545    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:09:10.300750 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:10.300726    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:09:29.522271 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522239    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:29.522784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522627    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="main"
Jul 08 17:09:29.522784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522642    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="main"
Jul 08 17:09:29.522784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522657    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="storage-initializer"
Jul 08 17:09:29.522784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522664    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="storage-initializer"
Jul 08 17:09:29.522784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.522726    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="13102fb4-9acb-4c99-bde3-d53167ba0f6f" containerName="main"
Jul 08 17:09:29.526952 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.526933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.529792 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.529759    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-dc4rb\""
Jul 08 17:09:29.529792 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.529776    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:09:29.530015 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.529796    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 08 17:09:29.530015 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.529760    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:09:29.541297 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.541269    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:29.614234 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614234 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614464 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wclvr\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614464 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614464 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614464 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614622 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614622 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.614622 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.614563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715492 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715492 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wclvr\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.715764 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.715704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.716135 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.716009    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.716188 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.716149    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.716225 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.716206    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.716270 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.716209    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.716346 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.716326    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.718190 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.718164    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.718497 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.718472    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.725354 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.725330    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wclvr\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.725475 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.725330    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-z6cd2\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:29.837905 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:29.837797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:30.004481 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.004452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:30.006588 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:09:30.006536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfe1eba_cfdd_4114_bd06_422e74dc2fe4.slice/crio-4d519915bc050b19a265ea3682c0a0944d9e3e695c9fed6a7a2753c65a05bdb4 WatchSource:0}: Error finding container 4d519915bc050b19a265ea3682c0a0944d9e3e695c9fed6a7a2753c65a05bdb4: Status 404 returned error can't find the container with id 4d519915bc050b19a265ea3682c0a0944d9e3e695c9fed6a7a2753c65a05bdb4
Jul 08 17:09:30.008878 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.008845    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 17:09:30.008970 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.008911    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 17:09:30.008970 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.008941    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 08 17:09:30.690999 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.690963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" event={"ID":"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4","Type":"ContainerStarted","Data":"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"}
Jul 08 17:09:30.691432 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.691020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" event={"ID":"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4","Type":"ContainerStarted","Data":"4d519915bc050b19a265ea3682c0a0944d9e3e695c9fed6a7a2753c65a05bdb4"}
Jul 08 17:09:30.721715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.721633    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" podStartSLOduration=1.721611419 podStartE2EDuration="1.721611419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:09:30.712712519 +0000 UTC m=+2120.930909048" watchObservedRunningTime="2026-07-08 17:09:30.721611419 +0000 UTC m=+2120.939807927"
Jul 08 17:09:30.838747 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:30.838713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:31.656509 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:31.656467    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:31.843307 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:31.843279    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:32.700743 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:32.700698    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" podUID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" containerName="istio-proxy" containerID="cri-o://b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51" gracePeriod=30
Jul 08 17:09:32.700969 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:32.700832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:32.701908 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:32.701882    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:37.958479 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:37.958445    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:38.091588 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.091808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091618    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.091808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.091808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.091808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.091808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091789    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.092199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091826    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.092199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091853    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.092199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.091897    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wclvr\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr\") pod \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\" (UID: \"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4\") "
Jul 08 17:09:38.092199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092085    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:38.092199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092103    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:38.092447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092236    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.092447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092244    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:38.092447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092256    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.092447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092271    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:38.092447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.092334    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:09:38.094053 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.094024    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:38.094180 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.094112    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 17:09:38.094180 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.094111    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:09:38.094339 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.094320    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr" (OuterVolumeSpecName: "kube-api-access-wclvr") pod "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" (UID: "1cfe1eba-cfdd-4114-bd06-422e74dc2fe4"). InnerVolumeSpecName "kube-api-access-wclvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:09:38.192975 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.192933    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.192975 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.192964    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.192975 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.192974    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wclvr\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-kube-api-access-wclvr\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.192975 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.192985    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.193248 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.192993    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.193248 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.193001    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.193248 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.193009    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:09:38.739841 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.739802    2559 generic.go:356] "Generic (PLEG): container finished" podID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" containerID="b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51" exitCode=0
Jul 08 17:09:38.740005 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.739848    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"
Jul 08 17:09:38.740005 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.739883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" event={"ID":"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4","Type":"ContainerDied","Data":"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"}
Jul 08 17:09:38.740005 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.739937    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2" event={"ID":"1cfe1eba-cfdd-4114-bd06-422e74dc2fe4","Type":"ContainerDied","Data":"4d519915bc050b19a265ea3682c0a0944d9e3e695c9fed6a7a2753c65a05bdb4"}
Jul 08 17:09:38.740005 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.739959    2559 scope.go:117] "RemoveContainer" containerID="b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"
Jul 08 17:09:38.752529 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.752511    2559 scope.go:117] "RemoveContainer" containerID="b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"
Jul 08 17:09:38.752869 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:09:38.752848    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51\": container with ID starting with b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51 not found: ID does not exist" containerID="b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"
Jul 08 17:09:38.752933 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.752879    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51"} err="failed to get container status \"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51\": rpc error: code = NotFound desc = could not find container \"b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51\": container with ID starting with b04bc783c26e5e3e585b3339a57fddbe062978c891667583787ff7c1e000ad51 not found: ID does not exist"
Jul 08 17:09:38.763608 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.763561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:38.768909 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:38.768884    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-z6cd2"]
Jul 08 17:09:40.244758 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:40.244730    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" path="/var/lib/kubelet/pods/1cfe1eba-cfdd-4114-bd06-422e74dc2fe4/volumes"
Jul 08 17:09:44.451814 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.451777    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:09:44.452216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.452137    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" containerName="istio-proxy"
Jul 08 17:09:44.452216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.452151    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" containerName="istio-proxy"
Jul 08 17:09:44.452291 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.452220    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfe1eba-cfdd-4114-bd06-422e74dc2fe4" containerName="istio-proxy"
Jul 08 17:09:44.455326 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.455307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.458044 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.458019    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:09:44.458188 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.458055    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:09:44.459393 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.459365    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:09:44.459393 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.459367    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 08 17:09:44.459558 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.459396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-4h2hq\""
Jul 08 17:09:44.459558 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.459376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:09:44.466360 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.466339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:09:44.551736 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.551696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.551736 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.551740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.551945 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.551776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4jp\" (UniqueName: \"kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.551945 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.551889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653251 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653214    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4jp\" (UniqueName: \"kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653746 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653721    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.653793 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.653729    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.656025 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.655995    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.663812 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.663788    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4jp\" (UniqueName: \"kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp\") pod \"router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.765338 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.765241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:44.908610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:44.908562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:09:44.910722 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:09:44.910692    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd4fce5_85dc_4fd2_b058_1997fb35cc3f.slice/crio-493a876439fd3f86dc8d0c6ed8046aa035c148d7a00dd7cb1386c2eb7364c8d9 WatchSource:0}: Error finding container 493a876439fd3f86dc8d0c6ed8046aa035c148d7a00dd7cb1386c2eb7364c8d9: Status 404 returned error can't find the container with id 493a876439fd3f86dc8d0c6ed8046aa035c148d7a00dd7cb1386c2eb7364c8d9
Jul 08 17:09:45.781634 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:45.781564    2559 generic.go:356] "Generic (PLEG): container finished" podID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerID="7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc" exitCode=0
Jul 08 17:09:45.782128 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:45.781724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" event={"ID":"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f","Type":"ContainerDied","Data":"7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc"}
Jul 08 17:09:45.782128 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:45.781793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" event={"ID":"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f","Type":"ContainerStarted","Data":"493a876439fd3f86dc8d0c6ed8046aa035c148d7a00dd7cb1386c2eb7364c8d9"}
Jul 08 17:09:46.788285 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:46.788250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" event={"ID":"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f","Type":"ContainerStarted","Data":"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"}
Jul 08 17:09:46.788735 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:46.788384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:09:46.809211 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:09:46.809146    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" podStartSLOduration=2.151930493 podStartE2EDuration="2.80912396s" totalImagesPullingTime="0s" totalInitContainerRuntime="657.193467ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:09:46.807674386 +0000 UTC m=+2137.025870916" watchObservedRunningTime="2026-07-08 17:09:46.80912396 +0000 UTC m=+2137.027320472"
Jul 08 17:10:17.795679 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:10:17.795648    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:11:58.573192 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.573158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:11:58.583218 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:11:58.583188    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:11:58.583218 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:11:58.583218 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:11:58.583218 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27" cmd=["/bin/sleep","15"]
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.583233    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" message="" err=<
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:11:58.583246    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="main" containerID="cri-o://aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"
Jul 08 17:11:58.583467 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.583283    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="main" containerID="cri-o://aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27" gracePeriod=60
Jul 08 17:11:58.833729 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.833669    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:11:58.973429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.973385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds\") pod \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") "
Jul 08 17:11:58.973429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.973432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh4jp\" (UniqueName: \"kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp\") pod \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") "
Jul 08 17:11:58.973688 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.973453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs\") pod \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") "
Jul 08 17:11:58.973688 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.973480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location\") pod \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\" (UID: \"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f\") "
Jul 08 17:11:58.973813 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.973727    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" (UID: "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:11:58.974234 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.974206    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" (UID: "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:11:58.975544 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.975528    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" (UID: "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:11:58.975678 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:58.975662    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp" (OuterVolumeSpecName: "kube-api-access-vh4jp") pod "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" (UID: "8cd4fce5-85dc-4fd2-b058-1997fb35cc3f"). InnerVolumeSpecName "kube-api-access-vh4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:11:59.074272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.074246    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:11:59.074272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.074269    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh4jp\" (UniqueName: \"kubernetes.io/projected/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kube-api-access-vh4jp\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:11:59.074272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.074279    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:11:59.074477 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.074287    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:11:59.431339 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.431309    2559 generic.go:356] "Generic (PLEG): container finished" podID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerID="aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27" exitCode=0
Jul 08 17:11:59.431510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.431359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"
Jul 08 17:11:59.431510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.431372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" event={"ID":"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f","Type":"ContainerDied","Data":"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"}
Jul 08 17:11:59.431510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.431411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt" event={"ID":"8cd4fce5-85dc-4fd2-b058-1997fb35cc3f","Type":"ContainerDied","Data":"493a876439fd3f86dc8d0c6ed8046aa035c148d7a00dd7cb1386c2eb7364c8d9"}
Jul 08 17:11:59.431510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.431430    2559 scope.go:117] "RemoveContainer" containerID="aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"
Jul 08 17:11:59.441298 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.441281    2559 scope.go:117] "RemoveContainer" containerID="7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc"
Jul 08 17:11:59.449062 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.449045    2559 scope.go:117] "RemoveContainer" containerID="aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"
Jul 08 17:11:59.449351 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:11:59.449329    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27\": container with ID starting with aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27 not found: ID does not exist" containerID="aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"
Jul 08 17:11:59.449444 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.449355    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27"} err="failed to get container status \"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27\": rpc error: code = NotFound desc = could not find container \"aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27\": container with ID starting with aefd9c879ca2f88ca91d1e201434304c471ab6e7bd8efce743b9c3a6db5b7c27 not found: ID does not exist"
Jul 08 17:11:59.449444 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.449372    2559 scope.go:117] "RemoveContainer" containerID="7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc"
Jul 08 17:11:59.449633 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:11:59.449615    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc\": container with ID starting with 7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc not found: ID does not exist" containerID="7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc"
Jul 08 17:11:59.449679 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.449638    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc"} err="failed to get container status \"7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc\": rpc error: code = NotFound desc = could not find container \"7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc\": container with ID starting with 7dbd2ce7f33fbf3444a9b2030305f9220c902db22a05d21e45c4c147627fd0dc not found: ID does not exist"
Jul 08 17:11:59.456068 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.456044    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:11:59.462751 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:11:59.462730    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-85b8956674-9vvqt"]
Jul 08 17:12:00.244445 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:00.244412    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" path="/var/lib/kubelet/pods/8cd4fce5-85dc-4fd2-b058-1997fb35cc3f/volumes"
Jul 08 17:12:23.709949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.709916    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:12:23.710442 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.710426    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="main"
Jul 08 17:12:23.710487 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.710446    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="main"
Jul 08 17:12:23.710487 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.710463    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="storage-initializer"
Jul 08 17:12:23.710487 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.710472    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="storage-initializer"
Jul 08 17:12:23.710858 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.710559    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd4fce5-85dc-4fd2-b058-1997fb35cc3f" containerName="main"
Jul 08 17:12:23.713867 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.713847    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.717872 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.717847    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:12:23.718013 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.717852    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:12:23.718013 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.717903    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-qf2xz\""
Jul 08 17:12:23.718760 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.718740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 08 17:12:23.718986 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.718966    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:12:23.718986 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.718980    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:12:23.727896 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.727877    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:12:23.757893 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.757868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdhl\" (UniqueName: \"kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.758037 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.757920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.758037 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.758015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.758157 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.758065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859001 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.858965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859171 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.859027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859171 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.859052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859171 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.859082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdhl\" (UniqueName: \"kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859412 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.859390    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.859412 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.859407    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.861691 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.861670    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:23.875458 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:23.875428    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdhl\" (UniqueName: \"kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:24.023775 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:24.023749    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:24.165088 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:24.165057    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:12:24.168561 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:12:24.168540    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d53328_0744_4bbe_8d2e_7985607fec0c.slice/crio-3f41690247a883343fb7f3ae57fa3d94b0323a426bee2521982903039898b2ee WatchSource:0}: Error finding container 3f41690247a883343fb7f3ae57fa3d94b0323a426bee2521982903039898b2ee: Status 404 returned error can't find the container with id 3f41690247a883343fb7f3ae57fa3d94b0323a426bee2521982903039898b2ee
Jul 08 17:12:24.555965 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:24.555902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerStarted","Data":"5ee0a6d673d34ff977a0c6f7392ead0c8bee1df78cffba7add5bd0852740a5ae"}
Jul 08 17:12:24.556151 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:24.555985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerStarted","Data":"3f41690247a883343fb7f3ae57fa3d94b0323a426bee2521982903039898b2ee"}
Jul 08 17:12:25.562271 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:25.562236    2559 generic.go:356] "Generic (PLEG): container finished" podID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerID="5ee0a6d673d34ff977a0c6f7392ead0c8bee1df78cffba7add5bd0852740a5ae" exitCode=0
Jul 08 17:12:25.562681 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:25.562323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerDied","Data":"5ee0a6d673d34ff977a0c6f7392ead0c8bee1df78cffba7add5bd0852740a5ae"}
Jul 08 17:12:26.571606 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:26.571554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerStarted","Data":"c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149"}
Jul 08 17:12:26.572041 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:26.571667    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:12:26.596925 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:26.596870    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" podStartSLOduration=2.925575951 podStartE2EDuration="3.596855578s" totalImagesPullingTime="0s" totalInitContainerRuntime="671.279627ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:12:26.594116011 +0000 UTC m=+2296.812312538" watchObservedRunningTime="2026-07-08 17:12:26.596855578 +0000 UTC m=+2296.815052088"
Jul 08 17:12:57.578818 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:12:57.578790    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:13:11.945841 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:11.945810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:13:11.949395 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:11.949376    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:11.954997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:11.954973    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-pbcjh\""
Jul 08 17:13:11.955196 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:11.955171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 08 17:13:11.968070 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:11.968046    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:13:12.055931 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.055898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.056080 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.055950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.056080 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.056035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.056229 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.056081    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxhz\" (UniqueName: \"kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157218 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxhz\" (UniqueName: \"kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157670 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157654    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.157718 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.157691    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.159948 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.159928    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.180416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.180385    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxhz\" (UniqueName: \"kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.258232 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.258162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:12.396078 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.396056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:13:12.398585 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:13:12.398535    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b879014_28b2_4add_896c_d0b3c7244646.slice/crio-108da9c2a996c5beeb2fe14a736cb32fd7a71d997b7a28d922b02d15cce9e84e WatchSource:0}: Error finding container 108da9c2a996c5beeb2fe14a736cb32fd7a71d997b7a28d922b02d15cce9e84e: Status 404 returned error can't find the container with id 108da9c2a996c5beeb2fe14a736cb32fd7a71d997b7a28d922b02d15cce9e84e
Jul 08 17:13:12.792409 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.792368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerStarted","Data":"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"}
Jul 08 17:13:12.792624 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:12.792420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerStarted","Data":"108da9c2a996c5beeb2fe14a736cb32fd7a71d997b7a28d922b02d15cce9e84e"}
Jul 08 17:13:13.799436 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:13.799404    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b879014-28b2-4add-896c-d0b3c7244646" containerID="0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20" exitCode=0
Jul 08 17:13:13.799865 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:13.799466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerDied","Data":"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"}
Jul 08 17:13:14.810127 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:14.810081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerStarted","Data":"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"}
Jul 08 17:13:14.810541 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:14.810202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:13:14.836933 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:14.836883    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" podStartSLOduration=3.212260932 podStartE2EDuration="3.836868737s" totalImagesPullingTime="0s" totalInitContainerRuntime="624.607805ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:13:14.834618148 +0000 UTC m=+2345.052814666" watchObservedRunningTime="2026-07-08 17:13:14.836868737 +0000 UTC m=+2345.055065244"
Jul 08 17:13:45.818104 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:13:45.818070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:14:10.319385 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:14:10.319355    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:14:10.331543 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:14:10.331523    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:15:48.390244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.390207    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:15:48.402062 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:15:48.402023    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:15:48.402062 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:15:48.402062 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:15:48.402062 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149" cmd=["/bin/sleep","15"]
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.402076    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" message="" err=<
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:15:48.402090    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="main" containerID="cri-o://c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149"
Jul 08 17:15:48.402412 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.402125    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="main" containerID="cri-o://c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149" gracePeriod=60
Jul 08 17:15:48.546016 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.545980    2559 generic.go:356] "Generic (PLEG): container finished" podID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerID="c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149" exitCode=0
Jul 08 17:15:48.546196 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.546060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerDied","Data":"c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149"}
Jul 08 17:15:48.668692 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.668659    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:15:48.785653 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.785600    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds\") pod \"41d53328-0744-4bbe-8d2e-7985607fec0c\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") "
Jul 08 17:15:48.785870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.785666    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gdhl\" (UniqueName: \"kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl\") pod \"41d53328-0744-4bbe-8d2e-7985607fec0c\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") "
Jul 08 17:15:48.785870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.785741    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs\") pod \"41d53328-0744-4bbe-8d2e-7985607fec0c\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") "
Jul 08 17:15:48.785870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.785806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location\") pod \"41d53328-0744-4bbe-8d2e-7985607fec0c\" (UID: \"41d53328-0744-4bbe-8d2e-7985607fec0c\") "
Jul 08 17:15:48.786079 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.785950    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "41d53328-0744-4bbe-8d2e-7985607fec0c" (UID: "41d53328-0744-4bbe-8d2e-7985607fec0c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:15:48.786216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.786191    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:15:48.786808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.786775    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41d53328-0744-4bbe-8d2e-7985607fec0c" (UID: "41d53328-0744-4bbe-8d2e-7985607fec0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:15:48.788483 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.788454    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41d53328-0744-4bbe-8d2e-7985607fec0c" (UID: "41d53328-0744-4bbe-8d2e-7985607fec0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:15:48.788483 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.788464    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl" (OuterVolumeSpecName: "kube-api-access-9gdhl") pod "41d53328-0744-4bbe-8d2e-7985607fec0c" (UID: "41d53328-0744-4bbe-8d2e-7985607fec0c"). InnerVolumeSpecName "kube-api-access-9gdhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:15:48.886782 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.886745    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d53328-0744-4bbe-8d2e-7985607fec0c-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:15:48.886782 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.886773    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdhl\" (UniqueName: \"kubernetes.io/projected/41d53328-0744-4bbe-8d2e-7985607fec0c-kube-api-access-9gdhl\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:15:48.886782 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:48.886785    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d53328-0744-4bbe-8d2e-7985607fec0c-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:15:49.552402 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.552365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr" event={"ID":"41d53328-0744-4bbe-8d2e-7985607fec0c","Type":"ContainerDied","Data":"3f41690247a883343fb7f3ae57fa3d94b0323a426bee2521982903039898b2ee"}
Jul 08 17:15:49.552868 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.552432    2559 scope.go:117] "RemoveContainer" containerID="c0ab49949111a2c2d2f54a4d41c5258655d175a94f3a5da6fd357ffc4cb98149"
Jul 08 17:15:49.552868 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.552382    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"
Jul 08 17:15:49.562343 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.562325    2559 scope.go:117] "RemoveContainer" containerID="5ee0a6d673d34ff977a0c6f7392ead0c8bee1df78cffba7add5bd0852740a5ae"
Jul 08 17:15:49.576980 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.576948    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:15:49.581048 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:49.581022    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schepcckr"]
Jul 08 17:15:50.245001 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:15:50.244965    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" path="/var/lib/kubelet/pods/41d53328-0744-4bbe-8d2e-7985607fec0c/volumes"
Jul 08 17:16:03.551239 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.551010    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:16:03.561940 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:16:03.561874    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:16:03.561940 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:16:03.561940 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:16:03.561940 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412" cmd=["/bin/sleep","15"]
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.561958    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" message="" err=<
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:16:03.561980    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="main" containerID="cri-o://ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"
Jul 08 17:16:03.562305 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.562047    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="main" containerID="cri-o://ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412" gracePeriod=60
Jul 08 17:16:03.824927 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.824901    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:16:03.919689 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.919648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location\") pod \"5b879014-28b2-4add-896c-d0b3c7244646\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") "
Jul 08 17:16:03.919891 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.919728    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds\") pod \"5b879014-28b2-4add-896c-d0b3c7244646\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") "
Jul 08 17:16:03.919891 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.919766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs\") pod \"5b879014-28b2-4add-896c-d0b3c7244646\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") "
Jul 08 17:16:03.919891 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.919799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsxhz\" (UniqueName: \"kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz\") pod \"5b879014-28b2-4add-896c-d0b3c7244646\" (UID: \"5b879014-28b2-4add-896c-d0b3c7244646\") "
Jul 08 17:16:03.920058 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.920031    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5b879014-28b2-4add-896c-d0b3c7244646" (UID: "5b879014-28b2-4add-896c-d0b3c7244646"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:03.920391 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.920370    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b879014-28b2-4add-896c-d0b3c7244646" (UID: "5b879014-28b2-4add-896c-d0b3c7244646"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:03.922789 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.922759    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b879014-28b2-4add-896c-d0b3c7244646" (UID: "5b879014-28b2-4add-896c-d0b3c7244646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:16:03.922917 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:03.922806    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz" (OuterVolumeSpecName: "kube-api-access-vsxhz") pod "5b879014-28b2-4add-896c-d0b3c7244646" (UID: "5b879014-28b2-4add-896c-d0b3c7244646"). InnerVolumeSpecName "kube-api-access-vsxhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:16:04.021173 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.021140    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.021173 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.021167    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5b879014-28b2-4add-896c-d0b3c7244646-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.021173 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.021177    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b879014-28b2-4add-896c-d0b3c7244646-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.021407 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.021187    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxhz\" (UniqueName: \"kubernetes.io/projected/5b879014-28b2-4add-896c-d0b3c7244646-kube-api-access-vsxhz\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.637750 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.637712    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b879014-28b2-4add-896c-d0b3c7244646" containerID="ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412" exitCode=0
Jul 08 17:16:04.638216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.637775    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"
Jul 08 17:16:04.638216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.637789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerDied","Data":"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"}
Jul 08 17:16:04.638216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.637845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb" event={"ID":"5b879014-28b2-4add-896c-d0b3c7244646","Type":"ContainerDied","Data":"108da9c2a996c5beeb2fe14a736cb32fd7a71d997b7a28d922b02d15cce9e84e"}
Jul 08 17:16:04.638216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.637864    2559 scope.go:117] "RemoveContainer" containerID="ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"
Jul 08 17:16:04.647215 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.647193    2559 scope.go:117] "RemoveContainer" containerID="0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"
Jul 08 17:16:04.655745 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.655723    2559 scope.go:117] "RemoveContainer" containerID="ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"
Jul 08 17:16:04.656059 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:16:04.656040    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412\": container with ID starting with ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412 not found: ID does not exist" containerID="ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"
Jul 08 17:16:04.656099 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.656073    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412"} err="failed to get container status \"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412\": rpc error: code = NotFound desc = could not find container \"ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412\": container with ID starting with ba404ebbedd791644a0e1bd22ddd673e689015254acd654bfe79f32813d6a412 not found: ID does not exist"
Jul 08 17:16:04.656099 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.656091    2559 scope.go:117] "RemoveContainer" containerID="0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"
Jul 08 17:16:04.656339 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:16:04.656319    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20\": container with ID starting with 0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20 not found: ID does not exist" containerID="0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"
Jul 08 17:16:04.656380 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.656350    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20"} err="failed to get container status \"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20\": rpc error: code = NotFound desc = could not find container \"0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20\": container with ID starting with 0bd747b2c59ef1786ad0fd4def866e6d0d0a1afccec141f7059f88fc5c89ef20 not found: ID does not exist"
Jul 08 17:16:04.662898 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.662872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:16:04.667896 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:04.667869    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rhxsb"]
Jul 08 17:16:06.244665 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:06.244630    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b879014-28b2-4add-896c-d0b3c7244646" path="/var/lib/kubelet/pods/5b879014-28b2-4add-896c-d0b3c7244646/volumes"
Jul 08 17:16:19.425288 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425244    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:16:19.425751 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425734    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="storage-initializer"
Jul 08 17:16:19.425751 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425749    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="storage-initializer"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425778    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="main"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425785    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="main"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425792    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="storage-initializer"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425798    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="storage-initializer"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425807    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="main"
Jul 08 17:16:19.425839 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425814    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="main"
Jul 08 17:16:19.426106 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425891    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d53328-0744-4bbe-8d2e-7985607fec0c" containerName="main"
Jul 08 17:16:19.426106 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.425900    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b879014-28b2-4add-896c-d0b3c7244646" containerName="main"
Jul 08 17:16:19.429100 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.429082    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.435333 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.435272    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:16:19.435333 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.435285    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-j4dlq\""
Jul 08 17:16:19.435516 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.435408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:16:19.436704 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.436648    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 08 17:16:19.436786 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.436704    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:16:19.436943 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.436925    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:16:19.447125 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.447095    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:16:19.564729 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.564688    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.564916 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.564767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.564916 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.564799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhg6\" (UniqueName: \"kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.564916 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.564821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666021 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.665979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666239 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.666048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666239 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.666077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhg6\" (UniqueName: \"kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666239 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.666097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666463 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.666438    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.666530 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.666452    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.668788 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.668770    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.676711 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.676645    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhg6\" (UniqueName: \"kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.739101 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.739060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:19.891604 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:19.891562    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:16:19.893888 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:16:19.893846    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8af1018_7899_4d33_b299_d1ac803b3d36.slice/crio-0898d4bfce9aa858de04185fc47d76379a8376f606e28470e67d098a6e9473aa WatchSource:0}: Error finding container 0898d4bfce9aa858de04185fc47d76379a8376f606e28470e67d098a6e9473aa: Status 404 returned error can't find the container with id 0898d4bfce9aa858de04185fc47d76379a8376f606e28470e67d098a6e9473aa
Jul 08 17:16:20.717723 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:20.717682    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerID="d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5" exitCode=0
Jul 08 17:16:20.718109 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:20.717774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" event={"ID":"f8af1018-7899-4d33-b299-d1ac803b3d36","Type":"ContainerDied","Data":"d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5"}
Jul 08 17:16:20.718109 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:20.717865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" event={"ID":"f8af1018-7899-4d33-b299-d1ac803b3d36","Type":"ContainerStarted","Data":"0898d4bfce9aa858de04185fc47d76379a8376f606e28470e67d098a6e9473aa"}
Jul 08 17:16:21.725046 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:21.725000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" event={"ID":"f8af1018-7899-4d33-b299-d1ac803b3d36","Type":"ContainerStarted","Data":"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"}
Jul 08 17:16:21.725492 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:21.725190    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:16:21.751985 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:21.751930    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" podStartSLOduration=2.070189407 podStartE2EDuration="2.751914493s" totalImagesPullingTime="0s" totalInitContainerRuntime="681.725086ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:16:21.749869917 +0000 UTC m=+2531.968066428" watchObservedRunningTime="2026-07-08 17:16:21.751914493 +0000 UTC m=+2531.970111001"
Jul 08 17:16:30.601210 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.601169    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:16:30.605377 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.605353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.608093 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.608068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 08 17:16:30.608216 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.608144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-hz8fj\""
Jul 08 17:16:30.616904 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.616874    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:16:30.766179 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.766146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.766402 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.766262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.766402 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.766289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.766402 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.766371    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszpx\" (UniqueName: \"kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.867831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.867738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.867831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.867771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.867831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.867824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszpx\" (UniqueName: \"kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.868140 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.867893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.868233 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.868208    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.868282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.868242    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.870567 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.870541    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.879596 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.879553    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszpx\" (UniqueName: \"kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx\") pod \"stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:30.915563 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:30.915527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:31.066748 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:31.066716    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:16:31.068707 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:16:31.068680    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a3d476_940e_4427_aae5_0316cf0c7505.slice/crio-5ec0af32869006e4e4eca3be1715a80cbdc2ee5396de6ba5482a658b41b07b68 WatchSource:0}: Error finding container 5ec0af32869006e4e4eca3be1715a80cbdc2ee5396de6ba5482a658b41b07b68: Status 404 returned error can't find the container with id 5ec0af32869006e4e4eca3be1715a80cbdc2ee5396de6ba5482a658b41b07b68
Jul 08 17:16:31.775652 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:31.775606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerStarted","Data":"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"}
Jul 08 17:16:31.776166 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:31.775671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerStarted","Data":"5ec0af32869006e4e4eca3be1715a80cbdc2ee5396de6ba5482a658b41b07b68"}
Jul 08 17:16:32.782439 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:32.782404    2559 generic.go:356] "Generic (PLEG): container finished" podID="62a3d476-940e-4427-aae5-0316cf0c7505" containerID="7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4" exitCode=0
Jul 08 17:16:32.782848 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:32.782488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerDied","Data":"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"}
Jul 08 17:16:33.789936 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:33.789891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerStarted","Data":"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"}
Jul 08 17:16:33.790326 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:33.789978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:16:33.811155 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:33.811094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" podStartSLOduration=3.1190077880000002 podStartE2EDuration="3.81105582s" totalImagesPullingTime="0s" totalInitContainerRuntime="692.048032ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:16:33.809508688 +0000 UTC m=+2544.027705209" watchObservedRunningTime="2026-07-08 17:16:33.81105582 +0000 UTC m=+2544.029252329"
Jul 08 17:16:52.731112 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:16:52.731080    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:17:04.796588 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:17:04.796537    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:19:10.349363 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:10.349330    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:19:10.364117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:10.364088    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:19:59.286909 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.286873    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:19:59.296944 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:19:59.296910    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:19:59.296944 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:19:59.296944 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:19:59.296944 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2" cmd=["/bin/sleep","15"]
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.296958    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" message="" err=<
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:19:59.296970    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="main" containerID="cri-o://a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"
Jul 08 17:19:59.297297 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.297018    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="main" containerID="cri-o://a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2" gracePeriod=60
Jul 08 17:19:59.549301 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.549237    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:19:59.697876 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.697839    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdhg6\" (UniqueName: \"kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6\") pod \"f8af1018-7899-4d33-b299-d1ac803b3d36\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") "
Jul 08 17:19:59.698085 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.697911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs\") pod \"f8af1018-7899-4d33-b299-d1ac803b3d36\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") "
Jul 08 17:19:59.698085 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.697977    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location\") pod \"f8af1018-7899-4d33-b299-d1ac803b3d36\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") "
Jul 08 17:19:59.698085 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.698012    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds\") pod \"f8af1018-7899-4d33-b299-d1ac803b3d36\" (UID: \"f8af1018-7899-4d33-b299-d1ac803b3d36\") "
Jul 08 17:19:59.698346 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.698318    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f8af1018-7899-4d33-b299-d1ac803b3d36" (UID: "f8af1018-7899-4d33-b299-d1ac803b3d36"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:19:59.698684 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.698660    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8af1018-7899-4d33-b299-d1ac803b3d36" (UID: "f8af1018-7899-4d33-b299-d1ac803b3d36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:19:59.700272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.700244    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8af1018-7899-4d33-b299-d1ac803b3d36" (UID: "f8af1018-7899-4d33-b299-d1ac803b3d36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:19:59.700272 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.700251    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6" (OuterVolumeSpecName: "kube-api-access-kdhg6") pod "f8af1018-7899-4d33-b299-d1ac803b3d36" (UID: "f8af1018-7899-4d33-b299-d1ac803b3d36"). InnerVolumeSpecName "kube-api-access-kdhg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:19:59.798870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.798835    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:19:59.798870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.798867    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhg6\" (UniqueName: \"kubernetes.io/projected/f8af1018-7899-4d33-b299-d1ac803b3d36-kube-api-access-kdhg6\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:19:59.798870 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.798878    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8af1018-7899-4d33-b299-d1ac803b3d36-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:19:59.799111 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.798887    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8af1018-7899-4d33-b299-d1ac803b3d36-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:19:59.802953 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.802880    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerID="a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2" exitCode=0
Jul 08 17:19:59.802953 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.802944    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"
Jul 08 17:19:59.803102 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.802946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" event={"ID":"f8af1018-7899-4d33-b299-d1ac803b3d36","Type":"ContainerDied","Data":"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"}
Jul 08 17:19:59.803102 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.803082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6" event={"ID":"f8af1018-7899-4d33-b299-d1ac803b3d36","Type":"ContainerDied","Data":"0898d4bfce9aa858de04185fc47d76379a8376f606e28470e67d098a6e9473aa"}
Jul 08 17:19:59.803183 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.803103    2559 scope.go:117] "RemoveContainer" containerID="a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"
Jul 08 17:19:59.812444 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.812428    2559 scope.go:117] "RemoveContainer" containerID="d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5"
Jul 08 17:19:59.820254 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.820240    2559 scope.go:117] "RemoveContainer" containerID="a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"
Jul 08 17:19:59.820523 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:19:59.820506    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2\": container with ID starting with a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2 not found: ID does not exist" containerID="a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"
Jul 08 17:19:59.820568 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.820532    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2"} err="failed to get container status \"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2\": rpc error: code = NotFound desc = could not find container \"a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2\": container with ID starting with a54c9e7db4237621ec2b40581bcc6835e12c4724a8bd97b50416bfac53afd0e2 not found: ID does not exist"
Jul 08 17:19:59.820568 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.820547    2559 scope.go:117] "RemoveContainer" containerID="d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5"
Jul 08 17:19:59.820919 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:19:59.820895    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5\": container with ID starting with d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5 not found: ID does not exist" containerID="d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5"
Jul 08 17:19:59.820963 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.820927    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5"} err="failed to get container status \"d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5\": rpc error: code = NotFound desc = could not find container \"d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5\": container with ID starting with d1a20cada3cdbb60e8d6e6d7d3fd549469e5b2f14a56a669573a87e4a97ebcf5 not found: ID does not exist"
Jul 08 17:19:59.828684 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.828662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:19:59.833691 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:19:59.833671    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-74789fhwf6"]
Jul 08 17:20:00.245109 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:00.245035    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" path="/var/lib/kubelet/pods/f8af1018-7899-4d33-b299-d1ac803b3d36/volumes"
Jul 08 17:20:02.111381 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.111350    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:20:02.120475 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:20:02.120440    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:20:02.120475 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:20:02.120475 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:20:02.120475 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af" cmd=["/bin/sleep","15"]
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.120491    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" message="" err=<
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:20:02.120503    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="main" containerID="cri-o://9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"
Jul 08 17:20:02.120803 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.120539    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="main" containerID="cri-o://9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af" gracePeriod=60
Jul 08 17:20:02.373474 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.373415    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:20:02.520829 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.520801    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs\") pod \"62a3d476-940e-4427-aae5-0316cf0c7505\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") "
Jul 08 17:20:02.520992 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.520842    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location\") pod \"62a3d476-940e-4427-aae5-0316cf0c7505\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") "
Jul 08 17:20:02.520992 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.520888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rszpx\" (UniqueName: \"kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx\") pod \"62a3d476-940e-4427-aae5-0316cf0c7505\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") "
Jul 08 17:20:02.520992 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.520917    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds\") pod \"62a3d476-940e-4427-aae5-0316cf0c7505\" (UID: \"62a3d476-940e-4427-aae5-0316cf0c7505\") "
Jul 08 17:20:02.521258 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.521232    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "62a3d476-940e-4427-aae5-0316cf0c7505" (UID: "62a3d476-940e-4427-aae5-0316cf0c7505"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:02.521663 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.521565    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62a3d476-940e-4427-aae5-0316cf0c7505" (UID: "62a3d476-940e-4427-aae5-0316cf0c7505"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:02.523068 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.523040    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62a3d476-940e-4427-aae5-0316cf0c7505" (UID: "62a3d476-940e-4427-aae5-0316cf0c7505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:20:02.523159 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.523090    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx" (OuterVolumeSpecName: "kube-api-access-rszpx") pod "62a3d476-940e-4427-aae5-0316cf0c7505" (UID: "62a3d476-940e-4427-aae5-0316cf0c7505"). InnerVolumeSpecName "kube-api-access-rszpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:20:02.622185 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.622155    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a3d476-940e-4427-aae5-0316cf0c7505-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:20:02.622185 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.622180    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:20:02.622185 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.622192    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszpx\" (UniqueName: \"kubernetes.io/projected/62a3d476-940e-4427-aae5-0316cf0c7505-kube-api-access-rszpx\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:20:02.622395 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.622200    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/62a3d476-940e-4427-aae5-0316cf0c7505-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:20:02.817930 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.817896    2559 generic.go:356] "Generic (PLEG): container finished" podID="62a3d476-940e-4427-aae5-0316cf0c7505" containerID="9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af" exitCode=0
Jul 08 17:20:02.818123 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.817940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerDied","Data":"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"}
Jul 08 17:20:02.818123 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.817952    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"
Jul 08 17:20:02.818123 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.817978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp" event={"ID":"62a3d476-940e-4427-aae5-0316cf0c7505","Type":"ContainerDied","Data":"5ec0af32869006e4e4eca3be1715a80cbdc2ee5396de6ba5482a658b41b07b68"}
Jul 08 17:20:02.818123 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.817996    2559 scope.go:117] "RemoveContainer" containerID="9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"
Jul 08 17:20:02.827719 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.827700    2559 scope.go:117] "RemoveContainer" containerID="7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"
Jul 08 17:20:02.835295 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.835276    2559 scope.go:117] "RemoveContainer" containerID="9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"
Jul 08 17:20:02.835543 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:20:02.835527    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af\": container with ID starting with 9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af not found: ID does not exist" containerID="9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"
Jul 08 17:20:02.835602 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.835551    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af"} err="failed to get container status \"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af\": rpc error: code = NotFound desc = could not find container \"9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af\": container with ID starting with 9eac4665cfe16513796c0737d43ed8e34d6c756a4a681d32d07852728b12d6af not found: ID does not exist"
Jul 08 17:20:02.835602 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.835585    2559 scope.go:117] "RemoveContainer" containerID="7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"
Jul 08 17:20:02.835820 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:20:02.835802    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4\": container with ID starting with 7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4 not found: ID does not exist" containerID="7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"
Jul 08 17:20:02.835867 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.835827    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4"} err="failed to get container status \"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4\": rpc error: code = NotFound desc = could not find container \"7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4\": container with ID starting with 7f1a3e65ff963fb944609556a8fd09532e6552fbbd178730b92ccf452b4180b4 not found: ID does not exist"
Jul 08 17:20:02.843267 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.843244    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:20:02.849072 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:02.849051    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7cb57d7b84-rn8hp"]
Jul 08 17:20:04.246409 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:04.246366    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" path="/var/lib/kubelet/pods/62a3d476-940e-4427-aae5-0316cf0c7505/volumes"
Jul 08 17:20:40.067323 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067291    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067660    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="main"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067673    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="main"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067684    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="storage-initializer"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="storage-initializer"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067696    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="storage-initializer"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067702    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="storage-initializer"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067721    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="main"
Jul 08 17:20:40.067767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067726    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="main"
Jul 08 17:20:40.068017 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067780    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a3d476-940e-4427-aae5-0316cf0c7505" containerName="main"
Jul 08 17:20:40.068017 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.067789    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8af1018-7899-4d33-b299-d1ac803b3d36" containerName="main"
Jul 08 17:20:40.071006 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.070989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.078871 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.078849    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:20:40.078976 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.078915    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:20:40.079867 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.079848    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:20:40.081478 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.081463    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 08 17:20:40.081787 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.081773    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-qfnnl\""
Jul 08 17:20:40.082466 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.082451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:20:40.091752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.091733    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:20:40.115329 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.115303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.115457 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.115351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s84vk\" (UniqueName: \"kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.115457 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.115433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.115457 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.115451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216143 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216313 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s84vk\" (UniqueName: \"kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216313 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216313 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216497 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216479    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.216554 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.216522    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.218781 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.218759    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.226911 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.226886    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s84vk\" (UniqueName: \"kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.380239 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.380147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:40.533601 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:40.533557    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:20:40.536093 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:20:40.536069    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068b30a5_e376_4615_9578_cd80cc33fe6b.slice/crio-37feb7baa717dd0d2c0a179133be325c8d7060c0fb628e2d1f479d7136b326db WatchSource:0}: Error finding container 37feb7baa717dd0d2c0a179133be325c8d7060c0fb628e2d1f479d7136b326db: Status 404 returned error can't find the container with id 37feb7baa717dd0d2c0a179133be325c8d7060c0fb628e2d1f479d7136b326db
Jul 08 17:20:41.003042 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:41.003004    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerStarted","Data":"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"}
Jul 08 17:20:41.003245 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:41.003064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerStarted","Data":"37feb7baa717dd0d2c0a179133be325c8d7060c0fb628e2d1f479d7136b326db"}
Jul 08 17:20:42.009670 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:42.009637    2559 generic.go:356] "Generic (PLEG): container finished" podID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerID="9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca" exitCode=0
Jul 08 17:20:42.010077 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:42.009728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerDied","Data":"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"}
Jul 08 17:20:43.016381 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:43.016347    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerStarted","Data":"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"}
Jul 08 17:20:43.016774 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:43.016408    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:20:43.042872 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:20:43.042813    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" podStartSLOduration=2.397902508 podStartE2EDuration="3.042794512s" totalImagesPullingTime="0s" totalInitContainerRuntime="644.892004ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:20:43.042215538 +0000 UTC m=+2793.260412059" watchObservedRunningTime="2026-07-08 17:20:43.042794512 +0000 UTC m=+2793.260991020"
Jul 08 17:21:14.023831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:21:14.023802    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:24:10.378405 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:10.378379    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:24:10.394966 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:10.394944    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:24:13.635759 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:13.635707    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:24:13.645516 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:24:13.645479    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:24:13.645516 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:24:13.645516 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:24:13.645516 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70" cmd=["/bin/sleep","15"]
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:13.645529    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" message="" err=<
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:24:13.645541    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="main" containerID="cri-o://90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"
Jul 08 17:24:13.645832 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:13.645592    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="main" containerID="cri-o://90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70" gracePeriod=60
Jul 08 17:24:13.899424 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:13.899369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:24:14.000408 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.000378    2559 generic.go:356] "Generic (PLEG): container finished" podID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerID="90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70" exitCode=0
Jul 08 17:24:14.000565 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.000422    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"
Jul 08 17:24:14.000565 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.000465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerDied","Data":"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"}
Jul 08 17:24:14.000565 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.000519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6" event={"ID":"068b30a5-e376-4615-9578-cd80cc33fe6b","Type":"ContainerDied","Data":"37feb7baa717dd0d2c0a179133be325c8d7060c0fb628e2d1f479d7136b326db"}
Jul 08 17:24:14.000565 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.000538    2559 scope.go:117] "RemoveContainer" containerID="90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"
Jul 08 17:24:14.010379 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.010362    2559 scope.go:117] "RemoveContainer" containerID="9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"
Jul 08 17:24:14.018079 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.018064    2559 scope.go:117] "RemoveContainer" containerID="90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"
Jul 08 17:24:14.018345 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:24:14.018327    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70\": container with ID starting with 90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70 not found: ID does not exist" containerID="90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"
Jul 08 17:24:14.018409 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.018353    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70"} err="failed to get container status \"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70\": rpc error: code = NotFound desc = could not find container \"90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70\": container with ID starting with 90b43f2e1915a87fcea2903dac997f5f140cb7703b49fbcb213f257fa2c8bf70 not found: ID does not exist"
Jul 08 17:24:14.018409 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.018373    2559 scope.go:117] "RemoveContainer" containerID="9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"
Jul 08 17:24:14.018613 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:24:14.018587    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca\": container with ID starting with 9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca not found: ID does not exist" containerID="9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"
Jul 08 17:24:14.018707 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.018615    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca"} err="failed to get container status \"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca\": rpc error: code = NotFound desc = could not find container \"9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca\": container with ID starting with 9ed76f9ff7b0c5ed701f6296c9b47673749b1920712557799c146ea0d54079ca not found: ID does not exist"
Jul 08 17:24:14.094334 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.094310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s84vk\" (UniqueName: \"kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk\") pod \"068b30a5-e376-4615-9578-cd80cc33fe6b\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") "
Jul 08 17:24:14.094446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.094343    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location\") pod \"068b30a5-e376-4615-9578-cd80cc33fe6b\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") "
Jul 08 17:24:14.094446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.094383    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds\") pod \"068b30a5-e376-4615-9578-cd80cc33fe6b\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") "
Jul 08 17:24:14.094446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.094431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs\") pod \"068b30a5-e376-4615-9578-cd80cc33fe6b\" (UID: \"068b30a5-e376-4615-9578-cd80cc33fe6b\") "
Jul 08 17:24:14.094721 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.094697    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "068b30a5-e376-4615-9578-cd80cc33fe6b" (UID: "068b30a5-e376-4615-9578-cd80cc33fe6b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:14.095043 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.095022    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "068b30a5-e376-4615-9578-cd80cc33fe6b" (UID: "068b30a5-e376-4615-9578-cd80cc33fe6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:14.096453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.096435    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "068b30a5-e376-4615-9578-cd80cc33fe6b" (UID: "068b30a5-e376-4615-9578-cd80cc33fe6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:24:14.096453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.096448    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk" (OuterVolumeSpecName: "kube-api-access-s84vk") pod "068b30a5-e376-4615-9578-cd80cc33fe6b" (UID: "068b30a5-e376-4615-9578-cd80cc33fe6b"). InnerVolumeSpecName "kube-api-access-s84vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:24:14.195012 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.194944    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/068b30a5-e376-4615-9578-cd80cc33fe6b-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:24:14.195012 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.194968    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s84vk\" (UniqueName: \"kubernetes.io/projected/068b30a5-e376-4615-9578-cd80cc33fe6b-kube-api-access-s84vk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:24:14.195012 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.194979    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:24:14.195012 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.195001    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/068b30a5-e376-4615-9578-cd80cc33fe6b-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:24:14.319510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.319480    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:24:14.326084 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:14.326062    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-68dbc8995xw9h6"]
Jul 08 17:24:16.244802 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:24:16.244760    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" path="/var/lib/kubelet/pods/068b30a5-e376-4615-9578-cd80cc33fe6b/volumes"
Jul 08 17:26:35.661715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.661612    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:26:35.662320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.662027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="main"
Jul 08 17:26:35.662320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.662039    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="main"
Jul 08 17:26:35.662320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.662052    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="storage-initializer"
Jul 08 17:26:35.662320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.662061    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="storage-initializer"
Jul 08 17:26:35.662320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.662118    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="068b30a5-e376-4615-9578-cd80cc33fe6b" containerName="main"
Jul 08 17:26:35.665170 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.665146    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.669191 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669166    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:26:35.669338 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669308    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-th4n7\""
Jul 08 17:26:35.669416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669388    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:26:35.669416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:26:35.669538 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669479    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:26:35.669616 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.669174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 08 17:26:35.678049 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.678019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:26:35.782235 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.782184    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njkn8\" (UniqueName: \"kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.782235 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.782250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.782481 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.782272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.782481 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.782370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.882918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.882861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.882918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.882929    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njkn8\" (UniqueName: \"kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.883142 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.882960    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.883142 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.882979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.883284 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.883265    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.883406 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.883386    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.885615 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.885596    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.896446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.896420    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njkn8\" (UniqueName: \"kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8\") pod \"tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:35.978090 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:35.977983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:36.134132 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:36.132377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:26:36.136852 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:26:36.136823    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f844ae0_632d_4cac_a883_4284024592ba.slice/crio-bb613a2807e31dc01398044ca6ea65366e8dfcac3f4bf9e57086b189ddd65937 WatchSource:0}: Error finding container bb613a2807e31dc01398044ca6ea65366e8dfcac3f4bf9e57086b189ddd65937: Status 404 returned error can't find the container with id bb613a2807e31dc01398044ca6ea65366e8dfcac3f4bf9e57086b189ddd65937
Jul 08 17:26:36.684672 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:36.684639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerStarted","Data":"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"}
Jul 08 17:26:36.685140 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:36.684687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerStarted","Data":"bb613a2807e31dc01398044ca6ea65366e8dfcac3f4bf9e57086b189ddd65937"}
Jul 08 17:26:37.690970 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:37.690934    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f844ae0-632d-4cac-a883-4284024592ba" containerID="a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee" exitCode=0
Jul 08 17:26:37.691369 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:37.691000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerDied","Data":"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"}
Jul 08 17:26:38.697544 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:38.697509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerStarted","Data":"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"}
Jul 08 17:26:38.697959 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:38.697644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:26:38.722352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:26:38.722296    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" podStartSLOduration=3.063710236 podStartE2EDuration="3.722280231s" totalImagesPullingTime="0s" totalInitContainerRuntime="658.569995ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:26:38.721108938 +0000 UTC m=+3148.939305446" watchObservedRunningTime="2026-07-08 17:26:38.722280231 +0000 UTC m=+3148.940476743"
Jul 08 17:27:09.704467 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:09.704438    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:27:26.246528 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.246498    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:27:26.250050 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.250035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.253027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.253006    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 08 17:27:26.253149 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.253060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-mwrmc\""
Jul 08 17:27:26.264265 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.264242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:27:26.302883 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.302854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.302889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkz8\" (UniqueName: \"kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.302984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303149 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.303097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303149 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.303136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303247 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.303180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.303292 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.303258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.403788 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.403766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.403918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.403805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.403918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.403827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkz8\" (UniqueName: \"kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.403918 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.403878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404074 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.403972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404074 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404074 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404261 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404238    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404323 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404294    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404369 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404329    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.404369 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.404361    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.406055 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.406030    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.406329 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.406306    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.413392 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.413367    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkz8\" (UniqueName: \"kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.506781 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.506682    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:27:26.510652 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.510631    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.513498 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.513477    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-fblbt\""
Jul 08 17:27:26.523108 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.523088    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:27:26.560420 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.560398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:27:26.606367 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.606342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.606503 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.606375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.606503 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.606403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfqb\" (UniqueName: \"kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.606503 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.606471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.706916 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.706858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:27:26.707271 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfqb\" (UniqueName: \"kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.707338 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.707438 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.707493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.707813 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707776    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.707875 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.707825    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.709978 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.709950    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.710508 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:27:26.710486    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2597c560_52e7_44c2_b6ee_3622f2140cf6.slice/crio-86fb7001dab6e61c171a61c2cf99a7add0cbb66c11c950e378d3b22375dd1977 WatchSource:0}: Error finding container 86fb7001dab6e61c171a61c2cf99a7add0cbb66c11c950e378d3b22375dd1977: Status 404 returned error can't find the container with id 86fb7001dab6e61c171a61c2cf99a7add0cbb66c11c950e378d3b22375dd1977
Jul 08 17:27:26.717276 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.717251    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfqb\" (UniqueName: \"kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.821949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.821921    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:26.933517 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.933474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerStarted","Data":"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"}
Jul 08 17:27:26.933720 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.933542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerStarted","Data":"86fb7001dab6e61c171a61c2cf99a7add0cbb66c11c950e378d3b22375dd1977"}
Jul 08 17:27:26.974721 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:26.974692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:27:26.977289 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:27:26.977260    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27892e4d_e641_4426_b5f8_1088faa05f54.slice/crio-2ebe38cf3433e3bda70a752d3ae7e8bd5fffe88bba330c6099638efc4e8c4c86 WatchSource:0}: Error finding container 2ebe38cf3433e3bda70a752d3ae7e8bd5fffe88bba330c6099638efc4e8c4c86: Status 404 returned error can't find the container with id 2ebe38cf3433e3bda70a752d3ae7e8bd5fffe88bba330c6099638efc4e8c4c86
Jul 08 17:27:27.940567 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:27.940532    2559 generic.go:356] "Generic (PLEG): container finished" podID="27892e4d-e641-4426-b5f8-1088faa05f54" containerID="471e7ddce19509356441fa3a206742c3233169c179db65afc44493e237967357" exitCode=0
Jul 08 17:27:27.941169 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:27.940631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" event={"ID":"27892e4d-e641-4426-b5f8-1088faa05f54","Type":"ContainerDied","Data":"471e7ddce19509356441fa3a206742c3233169c179db65afc44493e237967357"}
Jul 08 17:27:27.941169 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:27.940688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" event={"ID":"27892e4d-e641-4426-b5f8-1088faa05f54","Type":"ContainerStarted","Data":"2ebe38cf3433e3bda70a752d3ae7e8bd5fffe88bba330c6099638efc4e8c4c86"}
Jul 08 17:27:28.947580 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:28.947538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" event={"ID":"27892e4d-e641-4426-b5f8-1088faa05f54","Type":"ContainerStarted","Data":"3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af"}
Jul 08 17:27:28.947963 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:28.947739    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:27:28.972705 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:28.972638    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" podStartSLOduration=2.31188161 podStartE2EDuration="2.97261979s" totalImagesPullingTime="0s" totalInitContainerRuntime="660.73818ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:27:28.972252323 +0000 UTC m=+3199.190448842" watchObservedRunningTime="2026-07-08 17:27:28.97261979 +0000 UTC m=+3199.190816298"
Jul 08 17:27:30.960495 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:30.960462    2559 generic.go:356] "Generic (PLEG): container finished" podID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerID="769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922" exitCode=0
Jul 08 17:27:30.960876 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:30.960540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerDied","Data":"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"}
Jul 08 17:27:30.961847 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:30.961830    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:27:58.141308 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:58.141221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerStarted","Data":"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"}
Jul 08 17:27:58.176071 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:58.176007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.355823241 podStartE2EDuration="32.175984848s" totalImagesPullingTime="26.858776507s" totalInitContainerRuntime="3.9613851s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:27:58.172991179 +0000 UTC m=+3228.391187711" watchObservedRunningTime="2026-07-08 17:27:58.175984848 +0000 UTC m=+3228.394181357"
Jul 08 17:27:59.957234 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:27:59.957207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:28:54.029317 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.029276    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:28:54.041144 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:28:54.041096    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:28:54.041144 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:28:54.041144 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:28:54.041144 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3" cmd=["/bin/sleep","15"]
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.041169    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" message="" err=<
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:28:54.041187    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="main" containerID="cri-o://31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"
Jul 08 17:28:54.041479 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.041249    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="main" containerID="cri-o://31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3" gracePeriod=60
Jul 08 17:28:54.306104 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.306067    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:28:54.407293 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407262    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njkn8\" (UniqueName: \"kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8\") pod \"4f844ae0-632d-4cac-a883-4284024592ba\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") "
Jul 08 17:28:54.407491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407353    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds\") pod \"4f844ae0-632d-4cac-a883-4284024592ba\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") "
Jul 08 17:28:54.407491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407434    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs\") pod \"4f844ae0-632d-4cac-a883-4284024592ba\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") "
Jul 08 17:28:54.407491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location\") pod \"4f844ae0-632d-4cac-a883-4284024592ba\" (UID: \"4f844ae0-632d-4cac-a883-4284024592ba\") "
Jul 08 17:28:54.407706 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407631    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4f844ae0-632d-4cac-a883-4284024592ba" (UID: "4f844ae0-632d-4cac-a883-4284024592ba"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:28:54.407885 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.407854    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:28:54.408237 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.408215    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f844ae0-632d-4cac-a883-4284024592ba" (UID: "4f844ae0-632d-4cac-a883-4284024592ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:28:54.409650 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.409625    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f844ae0-632d-4cac-a883-4284024592ba" (UID: "4f844ae0-632d-4cac-a883-4284024592ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:28:54.410001 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.409980    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8" (OuterVolumeSpecName: "kube-api-access-njkn8") pod "4f844ae0-632d-4cac-a883-4284024592ba" (UID: "4f844ae0-632d-4cac-a883-4284024592ba"). InnerVolumeSpecName "kube-api-access-njkn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:28:54.448121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.448096    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f844ae0-632d-4cac-a883-4284024592ba" containerID="31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3" exitCode=0
Jul 08 17:28:54.448269 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.448147    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"
Jul 08 17:28:54.448269 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.448182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerDied","Data":"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"}
Jul 08 17:28:54.448269 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.448248    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv" event={"ID":"4f844ae0-632d-4cac-a883-4284024592ba","Type":"ContainerDied","Data":"bb613a2807e31dc01398044ca6ea65366e8dfcac3f4bf9e57086b189ddd65937"}
Jul 08 17:28:54.448434 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.448276    2559 scope.go:117] "RemoveContainer" containerID="31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"
Jul 08 17:28:54.458443 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.458427    2559 scope.go:117] "RemoveContainer" containerID="a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"
Jul 08 17:28:54.467339 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.467316    2559 scope.go:117] "RemoveContainer" containerID="31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"
Jul 08 17:28:54.467648 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:28:54.467625    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3\": container with ID starting with 31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3 not found: ID does not exist" containerID="31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"
Jul 08 17:28:54.467766 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.467653    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3"} err="failed to get container status \"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3\": rpc error: code = NotFound desc = could not find container \"31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3\": container with ID starting with 31e53143f997a6c7a841f300f769b054bc13b224468f333fe7ce3b5c7a9ce5f3 not found: ID does not exist"
Jul 08 17:28:54.467766 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.467671    2559 scope.go:117] "RemoveContainer" containerID="a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"
Jul 08 17:28:54.467904 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:28:54.467876    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee\": container with ID starting with a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee not found: ID does not exist" containerID="a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"
Jul 08 17:28:54.467904 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.467893    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee"} err="failed to get container status \"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee\": rpc error: code = NotFound desc = could not find container \"a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee\": container with ID starting with a04211fe44f7a3804775916dc5c51a9ea3bb2f7762c4076674bf61cc05cd70ee not found: ID does not exist"
Jul 08 17:28:54.471732 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.471706    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:28:54.480150 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.480115    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5d7755bf56-kb4bv"]
Jul 08 17:28:54.508450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.508412    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f844ae0-632d-4cac-a883-4284024592ba-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:28:54.508450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.508455    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f844ae0-632d-4cac-a883-4284024592ba-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:28:54.508680 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:54.508473    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njkn8\" (UniqueName: \"kubernetes.io/projected/4f844ae0-632d-4cac-a883-4284024592ba-kube-api-access-njkn8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:28:56.244400 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:28:56.244365    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f844ae0-632d-4cac-a883-4284024592ba" path="/var/lib/kubelet/pods/4f844ae0-632d-4cac-a883-4284024592ba/volumes"
Jul 08 17:29:10.410297 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:10.410271    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:29:10.428774 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:10.428745    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:29:19.989912 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.989869    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:29:19.990449 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.990430    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="main"
Jul 08 17:29:19.990493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.990455    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="main"
Jul 08 17:29:19.990528 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.990501    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="storage-initializer"
Jul 08 17:29:19.990528 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.990510    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="storage-initializer"
Jul 08 17:29:19.990643 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.990630    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f844ae0-632d-4cac-a883-4284024592ba" containerName="main"
Jul 08 17:29:19.997396 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:19.997365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.000219 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.000196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 08 17:29:20.005419 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.005389    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:29:20.012612 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012556    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012721 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012721 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012807 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012859 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg727\" (UniqueName: \"kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012924 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.012979 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.012954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114267 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114453 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114416    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114770 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114770 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg727\" (UniqueName: \"kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114770 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114707    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114770 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114759    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.114976 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.114865    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.115029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.115015    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.116784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.116761    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.117396 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.117373    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.124526 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.124493    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg727\" (UniqueName: \"kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727\") pod \"prestop-hook-test-kserve-5b7d7d79ff-52bt2\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.311350 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.311308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:20.465394 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.465358    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:29:20.466923 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:29:20.466895    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17343672_b4a4_4ab8_8188_6652d81cf62e.slice/crio-60d9a64f8c9fb6ddc17916b3d06d312c1f376e54fb08bae1a43b28e3918539bb WatchSource:0}: Error finding container 60d9a64f8c9fb6ddc17916b3d06d312c1f376e54fb08bae1a43b28e3918539bb: Status 404 returned error can't find the container with id 60d9a64f8c9fb6ddc17916b3d06d312c1f376e54fb08bae1a43b28e3918539bb
Jul 08 17:29:20.588744 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.588643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerStarted","Data":"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"}
Jul 08 17:29:20.588744 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:20.588711    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerStarted","Data":"60d9a64f8c9fb6ddc17916b3d06d312c1f376e54fb08bae1a43b28e3918539bb"}
Jul 08 17:29:24.611388 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:24.611350    2559 generic.go:356] "Generic (PLEG): container finished" podID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerID="e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca" exitCode=0
Jul 08 17:29:24.611924 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:24.611394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerDied","Data":"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"}
Jul 08 17:29:25.618869 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:25.618825    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerStarted","Data":"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"}
Jul 08 17:29:25.645551 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:25.645492    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podStartSLOduration=3.044729987 podStartE2EDuration="6.645473827s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.60074384s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:29:25.64386537 +0000 UTC m=+3315.862061905" watchObservedRunningTime="2026-07-08 17:29:25.645473827 +0000 UTC m=+3315.863670339"
Jul 08 17:29:26.312134 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:26.312099    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:30.312510 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.312478    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:29:30.314432 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.314396    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:29:30.400696 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.400599    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:29:30.410790 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:29:30.410753    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:29:30.410790 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:29:30.410790 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:29:30.410790 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af" cmd=["/bin/sleep","15"]
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.410809    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" message="" err=<
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:29:30.410825    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="main" containerID="cri-o://3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af"
Jul 08 17:29:30.411134 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.410877    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="main" containerID="cri-o://3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af" gracePeriod=60
Jul 08 17:29:30.658353 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.658318    2559 generic.go:356] "Generic (PLEG): container finished" podID="27892e4d-e641-4426-b5f8-1088faa05f54" containerID="3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af" exitCode=0
Jul 08 17:29:30.658529 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.658393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" event={"ID":"27892e4d-e641-4426-b5f8-1088faa05f54","Type":"ContainerDied","Data":"3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af"}
Jul 08 17:29:30.690527 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.690498    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:29:30.813820 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.813774    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hfqb\" (UniqueName: \"kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb\") pod \"27892e4d-e641-4426-b5f8-1088faa05f54\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") "
Jul 08 17:29:30.814037 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.813834    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs\") pod \"27892e4d-e641-4426-b5f8-1088faa05f54\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") "
Jul 08 17:29:30.814037 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.813971    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location\") pod \"27892e4d-e641-4426-b5f8-1088faa05f54\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") "
Jul 08 17:29:30.814159 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.814061    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds\") pod \"27892e4d-e641-4426-b5f8-1088faa05f54\" (UID: \"27892e4d-e641-4426-b5f8-1088faa05f54\") "
Jul 08 17:29:30.814618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.814535    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "27892e4d-e641-4426-b5f8-1088faa05f54" (UID: "27892e4d-e641-4426-b5f8-1088faa05f54"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:30.815057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.815024    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27892e4d-e641-4426-b5f8-1088faa05f54" (UID: "27892e4d-e641-4426-b5f8-1088faa05f54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:30.816298 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.816270    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27892e4d-e641-4426-b5f8-1088faa05f54" (UID: "27892e4d-e641-4426-b5f8-1088faa05f54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:29:30.816430 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.816405    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb" (OuterVolumeSpecName: "kube-api-access-9hfqb") pod "27892e4d-e641-4426-b5f8-1088faa05f54" (UID: "27892e4d-e641-4426-b5f8-1088faa05f54"). InnerVolumeSpecName "kube-api-access-9hfqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:29:30.915505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.915408    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hfqb\" (UniqueName: \"kubernetes.io/projected/27892e4d-e641-4426-b5f8-1088faa05f54-kube-api-access-9hfqb\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:29:30.915505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.915443    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27892e4d-e641-4426-b5f8-1088faa05f54-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:29:30.915505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.915459    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:29:30.915505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:30.915473    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27892e4d-e641-4426-b5f8-1088faa05f54-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:29:31.665176 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.665151    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"
Jul 08 17:29:31.665675 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.665182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf" event={"ID":"27892e4d-e641-4426-b5f8-1088faa05f54","Type":"ContainerDied","Data":"2ebe38cf3433e3bda70a752d3ae7e8bd5fffe88bba330c6099638efc4e8c4c86"}
Jul 08 17:29:31.665675 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.665250    2559 scope.go:117] "RemoveContainer" containerID="3a390ad16bdc401d3015931f255f341fb184179b64405d574de8dd4db71659af"
Jul 08 17:29:31.675831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.675807    2559 scope.go:117] "RemoveContainer" containerID="471e7ddce19509356441fa3a206742c3233169c179db65afc44493e237967357"
Jul 08 17:29:31.690169 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.690139    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:29:31.694324 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:31.694302    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schebbsmf"]
Jul 08 17:29:32.246120 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:32.246085    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" path="/var/lib/kubelet/pods/27892e4d-e641-4426-b5f8-1088faa05f54/volumes"
Jul 08 17:29:40.312058 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:40.312019    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:29:46.671963 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:46.671928    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:29:50.311751 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:29:50.311709    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:00.312083 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:00.312031    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:01.695469 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:01.695412    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="main" containerID="cri-o://b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2" gracePeriod=45
Jul 08 17:30:02.464771 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.464744    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:30:02.504385 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504311    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvkz8\" (UniqueName: \"kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504385 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504639 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504393    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504639 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504639 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504487    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504639 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504508    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504639 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504539    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home\") pod \"2597c560-52e7-44c2-b6ee-3622f2140cf6\" (UID: \"2597c560-52e7-44c2-b6ee-3622f2140cf6\") "
Jul 08 17:30:02.504931 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504677    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:02.504931 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.504834    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.505525 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.505488    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home" (OuterVolumeSpecName: "home") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:02.507266 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.507187    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm" (OuterVolumeSpecName: "dshm") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:02.507381 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.507324    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:30:02.508094 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.508068    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8" (OuterVolumeSpecName: "kube-api-access-jvkz8") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "kube-api-access-jvkz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:30:02.524050 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.524027    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:02.546650 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.546620    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2597c560-52e7-44c2-b6ee-3622f2140cf6" (UID: "2597c560-52e7-44c2-b6ee-3622f2140cf6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:02.605334 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605309    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2597c560-52e7-44c2-b6ee-3622f2140cf6-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.605334 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605334    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.605493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605346    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.605493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605358    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkz8\" (UniqueName: \"kubernetes.io/projected/2597c560-52e7-44c2-b6ee-3622f2140cf6-kube-api-access-jvkz8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.605493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605371    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.605493 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.605383    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2597c560-52e7-44c2-b6ee-3622f2140cf6-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:02.820662 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.820624    2559 generic.go:356] "Generic (PLEG): container finished" podID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerID="b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2" exitCode=0
Jul 08 17:30:02.821075 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.820680    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 08 17:30:02.821075 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.820690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerDied","Data":"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"}
Jul 08 17:30:02.821075 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.820742    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2597c560-52e7-44c2-b6ee-3622f2140cf6","Type":"ContainerDied","Data":"86fb7001dab6e61c171a61c2cf99a7add0cbb66c11c950e378d3b22375dd1977"}
Jul 08 17:30:02.821075 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.820774    2559 scope.go:117] "RemoveContainer" containerID="b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"
Jul 08 17:30:02.831180 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.831134    2559 scope.go:117] "RemoveContainer" containerID="769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"
Jul 08 17:30:02.841713 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.841691    2559 scope.go:117] "RemoveContainer" containerID="b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"
Jul 08 17:30:02.841975 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:30:02.841951    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2\": container with ID starting with b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2 not found: ID does not exist" containerID="b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"
Jul 08 17:30:02.842054 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.841989    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2"} err="failed to get container status \"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2\": rpc error: code = NotFound desc = could not find container \"b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2\": container with ID starting with b54789f8022d666afafad42c315a85f58af667a3adfd5baddb09c6f3d579b1f2 not found: ID does not exist"
Jul 08 17:30:02.842054 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.842013    2559 scope.go:117] "RemoveContainer" containerID="769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"
Jul 08 17:30:02.842288 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:30:02.842269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922\": container with ID starting with 769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922 not found: ID does not exist" containerID="769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"
Jul 08 17:30:02.842328 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.842294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922"} err="failed to get container status \"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922\": rpc error: code = NotFound desc = could not find container \"769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922\": container with ID starting with 769d3d1c317defed7794494d49f7d0daa1b6d2eb2ee6ea3b20572675ab8b8922 not found: ID does not exist"
Jul 08 17:30:02.850982 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.850954    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:30:02.858233 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:02.858209    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 08 17:30:04.245248 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:04.245216    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" path="/var/lib/kubelet/pods/2597c560-52e7-44c2-b6ee-3622f2140cf6/volumes"
Jul 08 17:30:10.311995 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:10.311942    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:20.312794 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:20.312754    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:25.944914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.944871    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:30:25.945502 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="storage-initializer"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945504    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="storage-initializer"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="storage-initializer"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945529    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="storage-initializer"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945550    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="main"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945558    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="main"
Jul 08 17:30:25.945618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="main"
Jul 08 17:30:25.945966 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945624    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="main"
Jul 08 17:30:25.945966 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945707    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="27892e4d-e641-4426-b5f8-1088faa05f54" containerName="main"
Jul 08 17:30:25.945966 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.945719    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2597c560-52e7-44c2-b6ee-3622f2140cf6" containerName="main"
Jul 08 17:30:25.949033 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.949008    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:25.953031 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.953008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-ntr6b\""
Jul 08 17:30:25.953195 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.953013    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 08 17:30:25.960480 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:25.960455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:30:26.008032 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.007997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqb88\" (UniqueName: \"kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.008203 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.008144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.008203 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.008182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.108748 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.108708    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.108748 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.108751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.109012 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.108787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqb88\" (UniqueName: \"kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.109147 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.109126    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.111423 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.111397    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.118851 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.118825    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqb88\" (UniqueName: \"kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.261069 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.260982    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.402984 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.402931    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:30:26.405334 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:30:26.405287    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bde377_24bc_45fd_9547_d8a5df69402c.slice/crio-08cabcb6707bd396ac2c9f70418ea216a45313fdff0a944cfd69b87edcaa0051 WatchSource:0}: Error finding container 08cabcb6707bd396ac2c9f70418ea216a45313fdff0a944cfd69b87edcaa0051: Status 404 returned error can't find the container with id 08cabcb6707bd396ac2c9f70418ea216a45313fdff0a944cfd69b87edcaa0051
Jul 08 17:30:26.950373 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.950338    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" event={"ID":"d9bde377-24bc-45fd-9547-d8a5df69402c","Type":"ContainerStarted","Data":"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"}
Jul 08 17:30:26.950818 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.950390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" event={"ID":"d9bde377-24bc-45fd-9547-d8a5df69402c","Type":"ContainerStarted","Data":"08cabcb6707bd396ac2c9f70418ea216a45313fdff0a944cfd69b87edcaa0051"}
Jul 08 17:30:26.950818 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.950516    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:26.973318 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:26.973266    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" podStartSLOduration=1.973249866 podStartE2EDuration="1.973249866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:30:26.972440431 +0000 UTC m=+3377.190636964" watchObservedRunningTime="2026-07-08 17:30:26.973249866 +0000 UTC m=+3377.191446375"
Jul 08 17:30:30.312775 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:30.312729    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:40.312355 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:40.312310    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:50.312740 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:50.312700    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:30:57.958042 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:57.958015    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:59.310747 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.310715    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:30:59.320088 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:30:59.320050    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:30:59.320088 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:30:59.320088 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:30:59.320088 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7" cmd=["/bin/sleep","15"]
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.320107    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" message="" err=<
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:30:59.320124    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerName="main" containerID="cri-o://eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"
Jul 08 17:30:59.320350 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.320181    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerName="main" containerID="cri-o://eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7" gracePeriod=60
Jul 08 17:30:59.577676 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.577650    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:30:59.608020 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.607996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs\") pod \"d9bde377-24bc-45fd-9547-d8a5df69402c\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") "
Jul 08 17:30:59.608191 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.608048    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqb88\" (UniqueName: \"kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88\") pod \"d9bde377-24bc-45fd-9547-d8a5df69402c\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") "
Jul 08 17:30:59.608191 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.608079    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds\") pod \"d9bde377-24bc-45fd-9547-d8a5df69402c\" (UID: \"d9bde377-24bc-45fd-9547-d8a5df69402c\") "
Jul 08 17:30:59.608495 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.608466    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d9bde377-24bc-45fd-9547-d8a5df69402c" (UID: "d9bde377-24bc-45fd-9547-d8a5df69402c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:30:59.610312 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.610281    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9bde377-24bc-45fd-9547-d8a5df69402c" (UID: "d9bde377-24bc-45fd-9547-d8a5df69402c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:30:59.610714 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.610689    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88" (OuterVolumeSpecName: "kube-api-access-dqb88") pod "d9bde377-24bc-45fd-9547-d8a5df69402c" (UID: "d9bde377-24bc-45fd-9547-d8a5df69402c"). InnerVolumeSpecName "kube-api-access-dqb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:30:59.709636 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.709602    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9bde377-24bc-45fd-9547-d8a5df69402c-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:59.709636 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.709629    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqb88\" (UniqueName: \"kubernetes.io/projected/d9bde377-24bc-45fd-9547-d8a5df69402c-kube-api-access-dqb88\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:30:59.709636 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:30:59.709639    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d9bde377-24bc-45fd-9547-d8a5df69402c-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:00.118702 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.118668    2559 generic.go:356] "Generic (PLEG): container finished" podID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerID="eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7" exitCode=0
Jul 08 17:31:00.118914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.118723    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"
Jul 08 17:31:00.118914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.118760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" event={"ID":"d9bde377-24bc-45fd-9547-d8a5df69402c","Type":"ContainerDied","Data":"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"}
Jul 08 17:31:00.118914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.118812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv" event={"ID":"d9bde377-24bc-45fd-9547-d8a5df69402c","Type":"ContainerDied","Data":"08cabcb6707bd396ac2c9f70418ea216a45313fdff0a944cfd69b87edcaa0051"}
Jul 08 17:31:00.118914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.118840    2559 scope.go:117] "RemoveContainer" containerID="eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"
Jul 08 17:31:00.130134 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.130117    2559 scope.go:117] "RemoveContainer" containerID="eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"
Jul 08 17:31:00.130416 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:31:00.130377    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7\": container with ID starting with eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7 not found: ID does not exist" containerID="eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"
Jul 08 17:31:00.130416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.130407    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7"} err="failed to get container status \"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7\": rpc error: code = NotFound desc = could not find container \"eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7\": container with ID starting with eef2babe3199a5bc06745190c3a3878f8b14efef2f6c6114054ff0dcf94898c7 not found: ID does not exist"
Jul 08 17:31:00.143410 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.143390    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:31:00.148866 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.148846    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfsqtmv"]
Jul 08 17:31:00.245372 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.245336    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" path="/var/lib/kubelet/pods/d9bde377-24bc-45fd-9547-d8a5df69402c/volumes"
Jul 08 17:31:00.311844 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:00.311802    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:31:10.311795 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:10.311757    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:31:20.322628 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:20.322598    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:31:20.330310 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:20.330281    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:31:25.027588 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.027542    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:31:25.056162 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.056125    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:31:25.056491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.056472    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerName="main"
Jul 08 17:31:25.056491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.056486    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerName="main"
Jul 08 17:31:25.056654 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.056558    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9bde377-24bc-45fd-9547-d8a5df69402c" containerName="main"
Jul 08 17:31:25.059690 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.059670    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.075021 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.074997    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:31:25.131954 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.131919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.131965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.131993    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.132071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.132120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.132143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.132373 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.132170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85bl\" (UniqueName: \"kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.232834 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.232834 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232945    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.232970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85bl\" (UniqueName: \"kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233428 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.233219    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233428 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.233286    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233428 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.233346    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.233428 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.233361    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.235441 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.235419    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.235731 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.235713    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.250057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.250030    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85bl\" (UniqueName: \"kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl\") pod \"prestop-hook-test-kserve-5b7d7d79ff-k79hx\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.369340 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.369247    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:25.512892 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:25.512836    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:31:25.515692 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:31:25.515661    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b1a6cd_4167_44e0_8ebe_d2ed9353c9d4.slice/crio-96072af57efffc960d43252afb01a91ec5736c7a37c2734f701f9fbcab4df801 WatchSource:0}: Error finding container 96072af57efffc960d43252afb01a91ec5736c7a37c2734f701f9fbcab4df801: Status 404 returned error can't find the container with id 96072af57efffc960d43252afb01a91ec5736c7a37c2734f701f9fbcab4df801
Jul 08 17:31:26.246142 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:26.246108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerStarted","Data":"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"}
Jul 08 17:31:26.246505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:26.246168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerStarted","Data":"96072af57efffc960d43252afb01a91ec5736c7a37c2734f701f9fbcab4df801"}
Jul 08 17:31:30.268057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:30.267971    2559 generic.go:356] "Generic (PLEG): container finished" podID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerID="383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474" exitCode=0
Jul 08 17:31:30.268407 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:30.268046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerDied","Data":"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"}
Jul 08 17:31:31.274834 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:31.274792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerStarted","Data":"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"}
Jul 08 17:31:31.299673 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:31.299621    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" podStartSLOduration=1.867353292 podStartE2EDuration="6.299607552s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.43225426s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:31:31.297766725 +0000 UTC m=+3441.515963248" watchObservedRunningTime="2026-07-08 17:31:31.299607552 +0000 UTC m=+3441.517804058"
Jul 08 17:31:31.370398 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:31.370357    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:35.369432 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:35.369393    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:31:35.370768 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:35.370700    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 08 17:31:40.051760 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.051715    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" containerID="cri-o://aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc" gracePeriod=45
Jul 08 17:31:40.323288 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.323199    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Jul 08 17:31:40.834826 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.834797    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:31:40.874349 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874260    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874349 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874320    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874375    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg727\" (UniqueName: \"kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs\") pod \"17343672-b4a4-4ab8-8188-6652d81cf62e\" (UID: \"17343672-b4a4-4ab8-8188-6652d81cf62e\") "
Jul 08 17:31:40.874992 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.874925    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home" (OuterVolumeSpecName: "home") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:40.875751 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.875306    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache" (OuterVolumeSpecName: "model-cache") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:40.878823 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.878787    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:31:40.878823 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.878796    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727" (OuterVolumeSpecName: "kube-api-access-jg727") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "kube-api-access-jg727". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:31:40.879093 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.879064    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm" (OuterVolumeSpecName: "dshm") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:40.890168 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.890144    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:40.940413 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.940343    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17343672-b4a4-4ab8-8188-6652d81cf62e" (UID: "17343672-b4a4-4ab8-8188-6652d81cf62e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:40.975375 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975340    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975375 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975370    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg727\" (UniqueName: \"kubernetes.io/projected/17343672-b4a4-4ab8-8188-6652d81cf62e-kube-api-access-jg727\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975375 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975382    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17343672-b4a4-4ab8-8188-6652d81cf62e-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975391    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975401    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975409    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:40.975635 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:40.975417    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17343672-b4a4-4ab8-8188-6652d81cf62e-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:31:41.331201 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.331158    2559 generic.go:356] "Generic (PLEG): container finished" podID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerID="aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc" exitCode=0
Jul 08 17:31:41.331658 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.331231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"
Jul 08 17:31:41.331658 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.331239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerDied","Data":"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"}
Jul 08 17:31:41.331658 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.331329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2" event={"ID":"17343672-b4a4-4ab8-8188-6652d81cf62e","Type":"ContainerDied","Data":"60d9a64f8c9fb6ddc17916b3d06d312c1f376e54fb08bae1a43b28e3918539bb"}
Jul 08 17:31:41.331658 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.331352    2559 scope.go:117] "RemoveContainer" containerID="aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"
Jul 08 17:31:41.341361 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.341341    2559 scope.go:117] "RemoveContainer" containerID="e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"
Jul 08 17:31:41.349948 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.349930    2559 scope.go:117] "RemoveContainer" containerID="aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"
Jul 08 17:31:41.350218 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:31:41.350200    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc\": container with ID starting with aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc not found: ID does not exist" containerID="aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"
Jul 08 17:31:41.350300 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.350226    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc"} err="failed to get container status \"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc\": rpc error: code = NotFound desc = could not find container \"aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc\": container with ID starting with aee548c72e012d5c50a1068fa1abc10f514be4a1d198a8af82ed56e0394e69cc not found: ID does not exist"
Jul 08 17:31:41.350300 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.350241    2559 scope.go:117] "RemoveContainer" containerID="e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"
Jul 08 17:31:41.350482 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:31:41.350461    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca\": container with ID starting with e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca not found: ID does not exist" containerID="e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"
Jul 08 17:31:41.350546 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.350488    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca"} err="failed to get container status \"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca\": rpc error: code = NotFound desc = could not find container \"e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca\": container with ID starting with e4690be44ef155f783ffca9327163f886454fe820d004aba4636cbeaa95cf0ca not found: ID does not exist"
Jul 08 17:31:41.361223 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.361188    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:31:41.363854 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:41.363832    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-52bt2"]
Jul 08 17:31:42.245105 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:42.245059    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" path="/var/lib/kubelet/pods/17343672-b4a4-4ab8-8188-6652d81cf62e/volumes"
Jul 08 17:31:45.370703 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:45.370652    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 08 17:31:55.370416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:55.370372    2559 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Jul 08 17:31:55.466132 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:31:55.466096    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:32:10.490072 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:10.490022    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main" containerID="cri-o://52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372" gracePeriod=45
Jul 08 17:32:12.247851 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.247822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5b7d7d79ff-k79hx_98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4/main/0.log"
Jul 08 17:32:12.250432 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.250415    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:32:12.339208 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339406 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339406 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339336    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339406 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l85bl\" (UniqueName: \"kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location\") pod \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\" (UID: \"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4\") "
Jul 08 17:32:12.339644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339564    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home" (OuterVolumeSpecName: "home") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:12.339878 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339710    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:12.339878 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339754    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:12.339878 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339769    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.339878 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.339791    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.341445 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.341423    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm" (OuterVolumeSpecName: "dshm") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:12.341739 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.341719    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl" (OuterVolumeSpecName: "kube-api-access-l85bl") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "kube-api-access-l85bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:32:12.341820 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.341803    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:32:12.398967 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.398935    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" (UID: "98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:12.440178 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.440149    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.440178 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.440176    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kserve-provision-location\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.440318 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.440189    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.440318 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.440204    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.440318 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.440218    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85bl\" (UniqueName: \"kubernetes.io/projected/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4-kube-api-access-l85bl\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:32:12.494935 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.494911    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5b7d7d79ff-k79hx_98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4/main/0.log"
Jul 08 17:32:12.495259 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.495239    2559 generic.go:356] "Generic (PLEG): container finished" podID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerID="52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372" exitCode=130
Jul 08 17:32:12.495327 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.495294    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"
Jul 08 17:32:12.495327 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.495302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerDied","Data":"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"}
Jul 08 17:32:12.495397 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.495351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx" event={"ID":"98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4","Type":"ContainerDied","Data":"96072af57efffc960d43252afb01a91ec5736c7a37c2734f701f9fbcab4df801"}
Jul 08 17:32:12.495397 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.495378    2559 scope.go:117] "RemoveContainer" containerID="52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"
Jul 08 17:32:12.504343 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.504330    2559 scope.go:117] "RemoveContainer" containerID="383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"
Jul 08 17:32:12.512274 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.512258    2559 scope.go:117] "RemoveContainer" containerID="52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"
Jul 08 17:32:12.512529 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:32:12.512512    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372\": container with ID starting with 52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372 not found: ID does not exist" containerID="52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"
Jul 08 17:32:12.512568 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.512536    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372"} err="failed to get container status \"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372\": rpc error: code = NotFound desc = could not find container \"52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372\": container with ID starting with 52341766f59050d51eb1d60ee4042e8b6dbabed99b8d9ed90f2ad89e40309372 not found: ID does not exist"
Jul 08 17:32:12.512568 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.512551    2559 scope.go:117] "RemoveContainer" containerID="383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"
Jul 08 17:32:12.512808 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:32:12.512791    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474\": container with ID starting with 383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474 not found: ID does not exist" containerID="383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"
Jul 08 17:32:12.512853 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.512815    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474"} err="failed to get container status \"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474\": rpc error: code = NotFound desc = could not find container \"383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474\": container with ID starting with 383b3ab64197fe100ff94c7cdf823c7f6ac6233e2167052203cfa9db672d6474 not found: ID does not exist"
Jul 08 17:32:12.535933 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.535902    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:32:12.540618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:12.540598    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5b7d7d79ff-k79hx"]
Jul 08 17:32:14.247344 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:14.247258    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" path="/var/lib/kubelet/pods/98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4/volumes"
Jul 08 17:32:17.029728 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.029687    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:32:17.030476 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030452    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030479    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030493    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="storage-initializer"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030502    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="storage-initializer"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030565    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030601    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main"
Jul 08 17:32:17.030610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030611    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="storage-initializer"
Jul 08 17:32:17.030898 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030620    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="storage-initializer"
Jul 08 17:32:17.030898 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030786    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="17343672-b4a4-4ab8-8188-6652d81cf62e" containerName="main"
Jul 08 17:32:17.030898 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.030809    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b1a6cd-4167-44e0-8ebe-d2ed9353c9d4" containerName="main"
Jul 08 17:32:17.036339 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.036312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.042632 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.042607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 08 17:32:17.043644 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.043627    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:32:17.043714 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.043631    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:32:17.045831 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.045814    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:32:17.052917 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.052896    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:32:17.177609 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.177763 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzjr\" (UniqueName: \"kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.177763 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.177763 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.177763 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.177957 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.177787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278733 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278747    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzjr\" (UniqueName: \"kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.278934 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.278874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.279217 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.279101    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.279217 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.279169    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.279334 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.279290    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.281262 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.281209    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.281375 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.281359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.295172 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.295145    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzjr\" (UniqueName: \"kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr\") pod \"rolling-upgrade-test-kserve-54bc76f55b-84gdc\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.346649 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.346616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:17.495498 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.495473    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:32:17.497742 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:32:17.497714    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b0d95f_810f_42e8_aa1a_1801400cc920.slice/crio-ad97105e6dba07d479fe95783007bf8ca2841557408297dd61c20d3d530b3e47 WatchSource:0}: Error finding container ad97105e6dba07d479fe95783007bf8ca2841557408297dd61c20d3d530b3e47: Status 404 returned error can't find the container with id ad97105e6dba07d479fe95783007bf8ca2841557408297dd61c20d3d530b3e47
Jul 08 17:32:17.522196 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:17.522158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" event={"ID":"c1b0d95f-810f-42e8-aa1a-1801400cc920","Type":"ContainerStarted","Data":"ad97105e6dba07d479fe95783007bf8ca2841557408297dd61c20d3d530b3e47"}
Jul 08 17:32:18.527989 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:18.527948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" event={"ID":"c1b0d95f-810f-42e8-aa1a-1801400cc920","Type":"ContainerStarted","Data":"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"}
Jul 08 17:32:19.346999 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:19.346959    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:27.347491 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:27.347454    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:27.360241 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:27.360218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:27.383342 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:27.383293    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" podStartSLOduration=11.383279903 podStartE2EDuration="11.383279903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:32:18.55377567 +0000 UTC m=+3488.771972179" watchObservedRunningTime="2026-07-08 17:32:27.383279903 +0000 UTC m=+3497.601476402"
Jul 08 17:32:27.583658 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:27.583629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:32:39.093911 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.093808    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:32:39.097567 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.097547    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.102151 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.102128    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-pdbf6\""
Jul 08 17:32:39.102281 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.102193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 08 17:32:39.115766 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.115743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:32:39.266125 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.266092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.266125 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.266129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs679\" (UniqueName: \"kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.266345 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.266160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.367099 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.367016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.367277 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.367239    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.367364 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.367305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs679\" (UniqueName: \"kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.367767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.367739    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.369709 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.369691    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.383379 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.383352    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs679\" (UniqueName: \"kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.408287 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.408261    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.558652 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.558629    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:32:39.560306 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:32:39.560280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dbe546_76f7_4d18_b727_c6c4b08b5c9a.slice/crio-6c2d368d7bea5086b01ad05698401a7faa9aae4e5a2a9a61ee37bddb335520d1 WatchSource:0}: Error finding container 6c2d368d7bea5086b01ad05698401a7faa9aae4e5a2a9a61ee37bddb335520d1: Status 404 returned error can't find the container with id 6c2d368d7bea5086b01ad05698401a7faa9aae4e5a2a9a61ee37bddb335520d1
Jul 08 17:32:39.631058 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.631017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" event={"ID":"34dbe546-76f7-4d18-b727-c6c4b08b5c9a","Type":"ContainerStarted","Data":"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"}
Jul 08 17:32:39.631200 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.631069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" event={"ID":"34dbe546-76f7-4d18-b727-c6c4b08b5c9a","Type":"ContainerStarted","Data":"6c2d368d7bea5086b01ad05698401a7faa9aae4e5a2a9a61ee37bddb335520d1"}
Jul 08 17:32:39.631200 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.631143    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:32:39.656073 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:32:39.656021    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" podStartSLOduration=0.65600428 podStartE2EDuration="656.00428ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:32:39.653872844 +0000 UTC m=+3509.872069367" watchObservedRunningTime="2026-07-08 17:32:39.65600428 +0000 UTC m=+3509.874200789"
Jul 08 17:33:03.496964 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:03.496925    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:33:10.636972 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:10.636946    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:33:12.467427 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.467395    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:33:12.476856 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:33:12.476821    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:33:12.476856 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:12.476856 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:12.476856 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b" cmd=["/bin/sleep","15"]
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.476870    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" message="" err=<
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:33:12.476885    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerName="main" containerID="cri-o://066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"
Jul 08 17:33:12.477088 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.476919    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerName="main" containerID="cri-o://066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b" gracePeriod=60
Jul 08 17:33:12.746176 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.746153    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:33:12.795015 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.794982    2559 generic.go:356] "Generic (PLEG): container finished" podID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerID="066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b" exitCode=0
Jul 08 17:33:12.795184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.795028    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"
Jul 08 17:33:12.795184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.795061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" event={"ID":"34dbe546-76f7-4d18-b727-c6c4b08b5c9a","Type":"ContainerDied","Data":"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"}
Jul 08 17:33:12.795184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.795109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f" event={"ID":"34dbe546-76f7-4d18-b727-c6c4b08b5c9a","Type":"ContainerDied","Data":"6c2d368d7bea5086b01ad05698401a7faa9aae4e5a2a9a61ee37bddb335520d1"}
Jul 08 17:33:12.795184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.795130    2559 scope.go:117] "RemoveContainer" containerID="066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"
Jul 08 17:33:12.803874 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.803853    2559 scope.go:117] "RemoveContainer" containerID="066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"
Jul 08 17:33:12.804145 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:33:12.804110    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b\": container with ID starting with 066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b not found: ID does not exist" containerID="066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"
Jul 08 17:33:12.804221 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.804156    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b"} err="failed to get container status \"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b\": rpc error: code = NotFound desc = could not find container \"066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b\": container with ID starting with 066ce6f1084ccf6b98fc7ac2d6a63c6ad8d4ac31504685a49359e41a30f5c27b not found: ID does not exist"
Jul 08 17:33:12.832440 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.832416    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs\") pod \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") "
Jul 08 17:33:12.832530 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.832461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs679\" (UniqueName: \"kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679\") pod \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") "
Jul 08 17:33:12.832587 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.832537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds\") pod \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\" (UID: \"34dbe546-76f7-4d18-b727-c6c4b08b5c9a\") "
Jul 08 17:33:12.832844 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.832825    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "34dbe546-76f7-4d18-b727-c6c4b08b5c9a" (UID: "34dbe546-76f7-4d18-b727-c6c4b08b5c9a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:12.834609 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.834560    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679" (OuterVolumeSpecName: "kube-api-access-xs679") pod "34dbe546-76f7-4d18-b727-c6c4b08b5c9a" (UID: "34dbe546-76f7-4d18-b727-c6c4b08b5c9a"). InnerVolumeSpecName "kube-api-access-xs679". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:33:12.834708 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.834628    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34dbe546-76f7-4d18-b727-c6c4b08b5c9a" (UID: "34dbe546-76f7-4d18-b727-c6c4b08b5c9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:33:12.933758 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.933728    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:12.933758 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.933751    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs679\" (UniqueName: \"kubernetes.io/projected/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-kube-api-access-xs679\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:12.933758 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:12.933762    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/34dbe546-76f7-4d18-b727-c6c4b08b5c9a-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:13.126716 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:13.126689    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:33:13.129819 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:13.129796    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87xj26f"]
Jul 08 17:33:14.245045 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:14.244974    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" path="/var/lib/kubelet/pods/34dbe546-76f7-4d18-b727-c6c4b08b5c9a/volumes"
Jul 08 17:33:18.518699 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.518661    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerName="main" containerID="cri-o://5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49" gracePeriod=45
Jul 08 17:33:18.573885 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.573840    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 08 17:33:18.769620 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.769538    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:33:18.826026 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.825993    2559 generic.go:356] "Generic (PLEG): container finished" podID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerID="5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49" exitCode=0
Jul 08 17:33:18.826184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.826033    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"
Jul 08 17:33:18.826184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.826047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" event={"ID":"c1b0d95f-810f-42e8-aa1a-1801400cc920","Type":"ContainerDied","Data":"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"}
Jul 08 17:33:18.826184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.826101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc" event={"ID":"c1b0d95f-810f-42e8-aa1a-1801400cc920","Type":"ContainerDied","Data":"ad97105e6dba07d479fe95783007bf8ca2841557408297dd61c20d3d530b3e47"}
Jul 08 17:33:18.826184 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.826130    2559 scope.go:117] "RemoveContainer" containerID="5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"
Jul 08 17:33:18.834358 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.834342    2559 scope.go:117] "RemoveContainer" containerID="5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"
Jul 08 17:33:18.834664 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:33:18.834633    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49\": container with ID starting with 5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49 not found: ID does not exist" containerID="5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"
Jul 08 17:33:18.834762 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.834665    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49"} err="failed to get container status \"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49\": rpc error: code = NotFound desc = could not find container \"5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49\": container with ID starting with 5f88e8cf5130dad04ad95fe7a785f9963b93262f70f9bd304772fa49bc095e49 not found: ID does not exist"
Jul 08 17:33:18.879665 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879642    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879765 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879686    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879765 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879765 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879762    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqzjr\" (UniqueName: \"kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879906 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879794    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879906 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879854    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir\") pod \"c1b0d95f-810f-42e8-aa1a-1801400cc920\" (UID: \"c1b0d95f-810f-42e8-aa1a-1801400cc920\") "
Jul 08 17:33:18.879906 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.879879    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:18.880064 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.880000    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home" (OuterVolumeSpecName: "home") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:18.880138 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.880119    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:18.880182 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.880141    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:18.880182 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.880143    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:18.881823 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.881799    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm" (OuterVolumeSpecName: "dshm") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:18.881917 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.881855    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr" (OuterVolumeSpecName: "kube-api-access-qqzjr") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "kube-api-access-qqzjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:33:18.881961 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.881938    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1b0d95f-810f-42e8-aa1a-1801400cc920" (UID: "c1b0d95f-810f-42e8-aa1a-1801400cc920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:33:18.980632 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.980607    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:18.980632 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.980629    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1b0d95f-810f-42e8-aa1a-1801400cc920-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:18.980775 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.980638    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzjr\" (UniqueName: \"kubernetes.io/projected/c1b0d95f-810f-42e8-aa1a-1801400cc920-kube-api-access-qqzjr\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:18.980775 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:18.980647    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1b0d95f-810f-42e8-aa1a-1801400cc920-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:33:19.151962 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:19.151937    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:33:19.157161 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:19.157142    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-84gdc"]
Jul 08 17:33:20.244875 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:20.244836    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" path="/var/lib/kubelet/pods/c1b0d95f-810f-42e8-aa1a-1801400cc920/volumes"
Jul 08 17:33:27.555808 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.555777    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-76d8d77597-mztr2"]
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556116    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerName="main"
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556127    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerName="main"
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556136    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerName="main"
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556142    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerName="main"
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556199    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b0d95f-810f-42e8-aa1a-1801400cc920" containerName="main"
Jul 08 17:33:27.556207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.556208    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="34dbe546-76f7-4d18-b727-c6c4b08b5c9a" containerName="main"
Jul 08 17:33:27.559200 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.559184    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.573504 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.573483    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-76d8d77597-mztr2"]
Jul 08 17:33:27.646320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.646295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-cert\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.646447 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.646340    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97gw\" (UniqueName: \"kubernetes.io/projected/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-kube-api-access-z97gw\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.746694 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.746662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97gw\" (UniqueName: \"kubernetes.io/projected/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-kube-api-access-z97gw\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.746869 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.746736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-cert\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.749183 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.749164    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-cert\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.757726 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.757704    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97gw\" (UniqueName: \"kubernetes.io/projected/b31bc5c2-26be-45f6-8215-cbdf59ae0d4c-kube-api-access-z97gw\") pod \"llmisvc-controller-manager-76d8d77597-mztr2\" (UID: \"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c\") " pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:27.868498 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:27.868423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:28.008454 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:28.008429    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-76d8d77597-mztr2"]
Jul 08 17:33:28.010667 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:33:28.010634    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb31bc5c2_26be_45f6_8215_cbdf59ae0d4c.slice/crio-50e66161461a08aa7809fc7ef77b31c70ddf2f6d6db6167f7687b062b1197a27 WatchSource:0}: Error finding container 50e66161461a08aa7809fc7ef77b31c70ddf2f6d6db6167f7687b062b1197a27: Status 404 returned error can't find the container with id 50e66161461a08aa7809fc7ef77b31c70ddf2f6d6db6167f7687b062b1197a27
Jul 08 17:33:28.012429 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:28.012410    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:33:28.873440 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:28.873411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2" event={"ID":"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c","Type":"ContainerStarted","Data":"ffc45dee0414e900547a9e728b79500c0ef0a512ac2b4f4cbaa03adeb70bb067"}
Jul 08 17:33:28.873820 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:28.873456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2" event={"ID":"b31bc5c2-26be-45f6-8215-cbdf59ae0d4c","Type":"ContainerStarted","Data":"50e66161461a08aa7809fc7ef77b31c70ddf2f6d6db6167f7687b062b1197a27"}
Jul 08 17:33:28.873820 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:28.873548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:48.582142 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.582089    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2" podStartSLOduration=20.965336174 podStartE2EDuration="21.582073929s" totalImagesPullingTime="616.737755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:28.900338162 +0000 UTC m=+3559.118534670" watchObservedRunningTime="2026-07-08 17:33:48.582073929 +0000 UTC m=+3578.800270435"
Jul 08 17:33:48.582885 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.582868    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:33:48.586379 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.586364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.589376 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.589348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:33:48.590624 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.590605    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 08 17:33:48.590715 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.590654    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:33:48.590776 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.590721    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:33:48.603488 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.603466    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:33:48.727441 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727407    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.727634 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsmr\" (UniqueName: \"kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.727634 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.727634 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.727759 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.727759 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.727711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.828998 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.828959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsmr\" (UniqueName: \"kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829160 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829656 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829566    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829656 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829622    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.829828 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.829666    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.831456 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.831430    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.831752 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.831735    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.839563 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.839508    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsmr\" (UniqueName: \"kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:48.897372 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:48.897350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:49.049114 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:49.045139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:33:49.974783 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:49.974746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" event={"ID":"9c787c58-14c6-48f4-938a-e09896d361ae","Type":"ContainerStarted","Data":"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"}
Jul 08 17:33:49.975173 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:49.974800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" event={"ID":"9c787c58-14c6-48f4-938a-e09896d361ae","Type":"ContainerStarted","Data":"794b2638eb59a89013a89917b8f6c346057c1cd877a520307e184853026f01d4"}
Jul 08 17:33:50.025606 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:50.025519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" podStartSLOduration=2.02549651 podStartE2EDuration="2.02549651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:50.024433983 +0000 UTC m=+3580.242630502" watchObservedRunningTime="2026-07-08 17:33:50.02549651 +0000 UTC m=+3580.243693020"
Jul 08 17:33:50.898247 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:50.898214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:58.897494 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:58.897460    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:58.909965 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:58.909939    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:59.029365 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:59.029342    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:33:59.880634 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:59.880596    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-76d8d77597-mztr2"
Jul 08 17:33:59.979192 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:59.979160    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 17:33:59.979631 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:33:59.979374    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" containerName="manager" containerID="cri-o://d380146885350a4088795f68c367406e5b0cb99be0c58f567978cd99854d4eea" gracePeriod=30
Jul 08 17:34:10.440669 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:10.440640    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:34:10.462854 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:10.462827    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:34:20.981452 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:20.981407    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.27:8081/readyz\": dial tcp 10.132.0.27:8081: connect: connection refused"
Jul 08 17:34:21.127560 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.127463    2559 generic.go:356] "Generic (PLEG): container finished" podID="8758a05d-4255-4769-807c-b812a1d88cc1" containerID="d380146885350a4088795f68c367406e5b0cb99be0c58f567978cd99854d4eea" exitCode=0
Jul 08 17:34:21.127560 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.127536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" event={"ID":"8758a05d-4255-4769-807c-b812a1d88cc1","Type":"ContainerDied","Data":"d380146885350a4088795f68c367406e5b0cb99be0c58f567978cd99854d4eea"}
Jul 08 17:34:21.429905 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.429879    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 17:34:21.503842 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.503806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") pod \"8758a05d-4255-4769-807c-b812a1d88cc1\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") "
Jul 08 17:34:21.504011 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.503904    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4qxj\" (UniqueName: \"kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj\") pod \"8758a05d-4255-4769-807c-b812a1d88cc1\" (UID: \"8758a05d-4255-4769-807c-b812a1d88cc1\") "
Jul 08 17:34:21.506191 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.506154    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj" (OuterVolumeSpecName: "kube-api-access-r4qxj") pod "8758a05d-4255-4769-807c-b812a1d88cc1" (UID: "8758a05d-4255-4769-807c-b812a1d88cc1"). InnerVolumeSpecName "kube-api-access-r4qxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:34:21.506300 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.506158    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert" (OuterVolumeSpecName: "cert") pod "8758a05d-4255-4769-807c-b812a1d88cc1" (UID: "8758a05d-4255-4769-807c-b812a1d88cc1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:34:21.604849 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.604813    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4qxj\" (UniqueName: \"kubernetes.io/projected/8758a05d-4255-4769-807c-b812a1d88cc1-kube-api-access-r4qxj\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:34:21.604849 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:21.604844    2559 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8758a05d-4255-4769-807c-b812a1d88cc1-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:34:22.133902 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.133874    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd"
Jul 08 17:34:22.134362 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.133920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-868c8c7759-p6tmd" event={"ID":"8758a05d-4255-4769-807c-b812a1d88cc1","Type":"ContainerDied","Data":"f226defc1f5327772d38712e7ca92583d6d69eea3b3f800d46a2e86795aaea0c"}
Jul 08 17:34:22.134362 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.133976    2559 scope.go:117] "RemoveContainer" containerID="d380146885350a4088795f68c367406e5b0cb99be0c58f567978cd99854d4eea"
Jul 08 17:34:22.158785 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.158754    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 17:34:22.162520 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.162495    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-868c8c7759-p6tmd"]
Jul 08 17:34:22.244901 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:22.244867    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" path="/var/lib/kubelet/pods/8758a05d-4255-4769-807c-b812a1d88cc1/volumes"
Jul 08 17:34:49.582282 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:34:49.582248    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:35:04.602930 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.602832    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" podUID="9c787c58-14c6-48f4-938a-e09896d361ae" containerName="main" containerID="cri-o://1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0" gracePeriod=45
Jul 08 17:35:04.868065 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.868007    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:35:04.952782 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.952949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.952949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.952949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952863    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.952949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952901    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsmr\" (UniqueName: \"kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.952949 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.952943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs\") pod \"9c787c58-14c6-48f4-938a-e09896d361ae\" (UID: \"9c787c58-14c6-48f4-938a-e09896d361ae\") "
Jul 08 17:35:04.953238 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.953073    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:35:04.953238 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.953088    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home" (OuterVolumeSpecName: "home") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:35:04.953238 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.953174    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:35:04.953357 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.953242    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:04.953357 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.953262    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:04.955063 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.955042    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm" (OuterVolumeSpecName: "dshm") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:35:04.955196 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.955173    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:35:04.955246 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:04.955200    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr" (OuterVolumeSpecName: "kube-api-access-kbsmr") pod "9c787c58-14c6-48f4-938a-e09896d361ae" (UID: "9c787c58-14c6-48f4-938a-e09896d361ae"). InnerVolumeSpecName "kube-api-access-kbsmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:35:05.054130 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.054101    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c787c58-14c6-48f4-938a-e09896d361ae-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:05.054130 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.054125    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:05.054130 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.054134    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c787c58-14c6-48f4-938a-e09896d361ae-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:05.054338 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.054143    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsmr\" (UniqueName: \"kubernetes.io/projected/9c787c58-14c6-48f4-938a-e09896d361ae-kube-api-access-kbsmr\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:35:05.338140 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.338102    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c787c58-14c6-48f4-938a-e09896d361ae" containerID="1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0" exitCode=0
Jul 08 17:35:05.338140 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.338143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" event={"ID":"9c787c58-14c6-48f4-938a-e09896d361ae","Type":"ContainerDied","Data":"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"}
Jul 08 17:35:05.338370 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.338154    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"
Jul 08 17:35:05.338370 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.338187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh" event={"ID":"9c787c58-14c6-48f4-938a-e09896d361ae","Type":"ContainerDied","Data":"794b2638eb59a89013a89917b8f6c346057c1cd877a520307e184853026f01d4"}
Jul 08 17:35:05.338370 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.338208    2559 scope.go:117] "RemoveContainer" containerID="1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"
Jul 08 17:35:05.348454 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.348432    2559 scope.go:117] "RemoveContainer" containerID="1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"
Jul 08 17:35:05.349770 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:35:05.349739    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0\": container with ID starting with 1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0 not found: ID does not exist" containerID="1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"
Jul 08 17:35:05.349884 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.349778    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0"} err="failed to get container status \"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0\": rpc error: code = NotFound desc = could not find container \"1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0\": container with ID starting with 1dcb74eaee4b7eb22518b212f6285e060be66de79e64397e46b82a54bceb0eb0 not found: ID does not exist"
Jul 08 17:35:05.362352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.362325    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:35:05.364876 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:05.364856    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-jlqwh"]
Jul 08 17:35:06.245565 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:06.245533    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c787c58-14c6-48f4-938a-e09896d361ae" path="/var/lib/kubelet/pods/9c787c58-14c6-48f4-938a-e09896d361ae/volumes"
Jul 08 17:35:20.016401 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.016364    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"]
Jul 08 17:35:20.017029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.016896    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" containerName="manager"
Jul 08 17:35:20.017029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.016917    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" containerName="manager"
Jul 08 17:35:20.017029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.016941    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c787c58-14c6-48f4-938a-e09896d361ae" containerName="main"
Jul 08 17:35:20.017029 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.016951    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c787c58-14c6-48f4-938a-e09896d361ae" containerName="main"
Jul 08 17:35:20.017286 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.017080    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c787c58-14c6-48f4-938a-e09896d361ae" containerName="main"
Jul 08 17:35:20.017286 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.017099    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8758a05d-4255-4769-807c-b812a1d88cc1" containerName="manager"
Jul 08 17:35:20.022535 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.022513    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.025589 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.025549    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 17:35:20.026908 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.026820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 17:35:20.026908 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.026844    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:35:20.026908 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.026843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-6cjq8\""
Jul 08 17:35:20.027124 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.026913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 08 17:35:20.035078 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.035053    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"]
Jul 08 17:35:20.073235 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.073358 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.073358 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.073446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.073446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.073446 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.073417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vcr\" (UniqueName: \"kubernetes.io/projected/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-kube-api-access-94vcr\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174275 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174275 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vcr\" (UniqueName: \"kubernetes.io/projected/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-kube-api-access-94vcr\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174275 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174275 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174492 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174553 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174523    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174638 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174612    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.174679 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.174647    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.176744 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.176721    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.176972 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.176957    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.186672 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.186647    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vcr\" (UniqueName: \"kubernetes.io/projected/1eaa996a-363c-4232-abfd-bfcf8a0a1aa2-kube-api-access-94vcr\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx\" (UID: \"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.333158 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.333124    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:20.479757 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:20.479731    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"]
Jul 08 17:35:20.482307 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:35:20.482279    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eaa996a_363c_4232_abfd_bfcf8a0a1aa2.slice/crio-169e94fda99ef10f8af5d94fa5916ed409aa848e62576587d219f4e45f66cc85 WatchSource:0}: Error finding container 169e94fda99ef10f8af5d94fa5916ed409aa848e62576587d219f4e45f66cc85: Status 404 returned error can't find the container with id 169e94fda99ef10f8af5d94fa5916ed409aa848e62576587d219f4e45f66cc85
Jul 08 17:35:21.416628 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:21.416594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx" event={"ID":"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2","Type":"ContainerStarted","Data":"0fbe4c84ce0edbef0e28e54622da9d6064fe832628cbba687d67edf284c332d9"}
Jul 08 17:35:21.417074 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:21.416644    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx" event={"ID":"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2","Type":"ContainerStarted","Data":"169e94fda99ef10f8af5d94fa5916ed409aa848e62576587d219f4e45f66cc85"}
Jul 08 17:35:21.417074 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:21.416681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:22.423171 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:22.423140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx" event={"ID":"1eaa996a-363c-4232-abfd-bfcf8a0a1aa2","Type":"ContainerStarted","Data":"ddaa99df2425a55be997512cfa9222e89e06842056b322465cbd0d4a8280c4c9"}
Jul 08 17:35:22.450506 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:22.450457    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx" podStartSLOduration=2.7250185030000003 podStartE2EDuration="3.450441979s" totalImagesPullingTime="725.423476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:35:22.447751366 +0000 UTC m=+3672.665947886" watchObservedRunningTime="2026-07-08 17:35:22.450441979 +0000 UTC m=+3672.668638488"
Jul 08 17:35:23.333245 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:23.333209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:28.236336 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.236299    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:35:28.240674 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.240647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.243690 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.243667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 08 17:35:28.243826 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.243801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-9hhnt\""
Jul 08 17:35:28.252427 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.252406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:35:28.339787 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.339755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.339902 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.339796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6f48\" (UniqueName: \"kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.339902 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.339831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.339981 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.339915    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.340018 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.339975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441000 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.440972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441172 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441172 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6f48\" (UniqueName: \"kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441307 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441271    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441391 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441470 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441451    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441762 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441742    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.441987 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.441968    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.443767 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.443748    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.452131 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.452104    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6f48\" (UniqueName: \"kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5677kqnn\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.551212 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.551187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:28.693320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:28.693293    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:35:28.695193 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:35:28.695165    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63a81e0_9344_4bae_9e83_11c85afadf41.slice/crio-7c98bef3cc5f9e5b71d5d080efb3f7cf014576c7b564f121543fd47045dba3dd WatchSource:0}: Error finding container 7c98bef3cc5f9e5b71d5d080efb3f7cf014576c7b564f121543fd47045dba3dd: Status 404 returned error can't find the container with id 7c98bef3cc5f9e5b71d5d080efb3f7cf014576c7b564f121543fd47045dba3dd
Jul 08 17:35:29.456610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:29.456559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" event={"ID":"f63a81e0-9344-4bae-9e83-11c85afadf41","Type":"ContainerStarted","Data":"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"}
Jul 08 17:35:29.457077 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:29.456617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" event={"ID":"f63a81e0-9344-4bae-9e83-11c85afadf41","Type":"ContainerStarted","Data":"7c98bef3cc5f9e5b71d5d080efb3f7cf014576c7b564f121543fd47045dba3dd"}
Jul 08 17:35:29.457077 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:29.456649    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:35:29.485207 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:29.485165    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" podStartSLOduration=1.485151587 podStartE2EDuration="1.485151587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:35:29.48459418 +0000 UTC m=+3679.702790683" watchObservedRunningTime="2026-07-08 17:35:29.485151587 +0000 UTC m=+3679.703348096"
Jul 08 17:35:30.334279 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:30.334245    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:30.346837 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:30.346813    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:30.473797 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:30.473728    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:35:31.480504 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:35:31.480472    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx"
Jul 08 17:36:00.463539 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:00.463508    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:36:01.655486 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.655451    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:36:01.665199 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:36:01.665164    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:36:01.665199 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:36:01.665199 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:36:01.665199 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a" cmd=["/bin/sleep","15"]
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.665210    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" message="" err=<
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:36:01.665223    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerName="main" containerID="cri-o://acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"
Jul 08 17:36:01.665505 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.665256    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerName="main" containerID="cri-o://acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a" gracePeriod=60
Jul 08 17:36:01.916035 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.915971    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:36:01.920620 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920600    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds\") pod \"f63a81e0-9344-4bae-9e83-11c85afadf41\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") "
Jul 08 17:36:01.920725 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920641    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6f48\" (UniqueName: \"kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48\") pod \"f63a81e0-9344-4bae-9e83-11c85afadf41\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") "
Jul 08 17:36:01.920725 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920712    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache\") pod \"f63a81e0-9344-4bae-9e83-11c85afadf41\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") "
Jul 08 17:36:01.920843 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920752    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs\") pod \"f63a81e0-9344-4bae-9e83-11c85afadf41\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") "
Jul 08 17:36:01.920843 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920812    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache\") pod \"f63a81e0-9344-4bae-9e83-11c85afadf41\" (UID: \"f63a81e0-9344-4bae-9e83-11c85afadf41\") "
Jul 08 17:36:01.920954 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920887    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f63a81e0-9344-4bae-9e83-11c85afadf41" (UID: "f63a81e0-9344-4bae-9e83-11c85afadf41"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:01.921008 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.920948    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "f63a81e0-9344-4bae-9e83-11c85afadf41" (UID: "f63a81e0-9344-4bae-9e83-11c85afadf41"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:01.921096 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.921079    2559 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-cachi2-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:36:01.921148 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.921104    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:36:01.921148 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.921109    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f63a81e0-9344-4bae-9e83-11c85afadf41" (UID: "f63a81e0-9344-4bae-9e83-11c85afadf41"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:01.922814 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.922794    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48" (OuterVolumeSpecName: "kube-api-access-t6f48") pod "f63a81e0-9344-4bae-9e83-11c85afadf41" (UID: "f63a81e0-9344-4bae-9e83-11c85afadf41"). InnerVolumeSpecName "kube-api-access-t6f48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:36:01.922950 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:01.922934    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f63a81e0-9344-4bae-9e83-11c85afadf41" (UID: "f63a81e0-9344-4bae-9e83-11c85afadf41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:36:02.022238 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.022211    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f63a81e0-9344-4bae-9e83-11c85afadf41-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:36:02.022238 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.022234    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f63a81e0-9344-4bae-9e83-11c85afadf41-tokenizer-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:36:02.022416 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.022245    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6f48\" (UniqueName: \"kubernetes.io/projected/f63a81e0-9344-4bae-9e83-11c85afadf41-kube-api-access-t6f48\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:36:02.626277 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.626240    2559 generic.go:356] "Generic (PLEG): container finished" podID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerID="acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a" exitCode=0
Jul 08 17:36:02.626450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.626288    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"
Jul 08 17:36:02.626450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.626320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" event={"ID":"f63a81e0-9344-4bae-9e83-11c85afadf41","Type":"ContainerDied","Data":"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"}
Jul 08 17:36:02.626450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.626369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn" event={"ID":"f63a81e0-9344-4bae-9e83-11c85afadf41","Type":"ContainerDied","Data":"7c98bef3cc5f9e5b71d5d080efb3f7cf014576c7b564f121543fd47045dba3dd"}
Jul 08 17:36:02.626450 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.626388    2559 scope.go:117] "RemoveContainer" containerID="acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"
Jul 08 17:36:02.635519 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.635499    2559 scope.go:117] "RemoveContainer" containerID="acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"
Jul 08 17:36:02.635829 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:36:02.635809    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a\": container with ID starting with acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a not found: ID does not exist" containerID="acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"
Jul 08 17:36:02.635903 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.635840    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a"} err="failed to get container status \"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a\": rpc error: code = NotFound desc = could not find container \"acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a\": container with ID starting with acf54f4c07b9d83d5be593dab9e884124500e6da08be6630d05c72852732566a not found: ID does not exist"
Jul 08 17:36:02.645114 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.645091    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:36:02.652069 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:02.652048    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5677kqnn"]
Jul 08 17:36:04.244993 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:36:04.244959    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" path="/var/lib/kubelet/pods/f63a81e0-9344-4bae-9e83-11c85afadf41/volumes"
Jul 08 17:39:10.469537 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:39:10.469469    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:39:10.491969 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:39:10.491948    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:44:10.501794 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:44:10.501690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:44:10.521082 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:44:10.521062    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:49:10.531016 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:49:10.530991    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:49:10.551853 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:49:10.551830    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:54:10.562623 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:10.562505    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:54:10.581495 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:10.581476    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:54:17.832605 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:17.832530    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:54:17.844924 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:17.844894    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:54:25.730389 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.730354    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:54:25.730849 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.730833    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerName="main"
Jul 08 17:54:25.730909 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.730851    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerName="main"
Jul 08 17:54:25.730909 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.730908    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63a81e0-9344-4bae-9e83-11c85afadf41" containerName="main"
Jul 08 17:54:25.733961 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.733945    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.738432 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.738404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-46bjz\""
Jul 08 17:54:25.738562 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.738475    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 08 17:54:25.744655 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.744635    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:54:25.784270 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.784242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4n8m\" (UniqueName: \"kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.784390 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.784283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.784390 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.784351    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.884788 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.884763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4n8m\" (UniqueName: \"kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.884951 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.884816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.884951 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.884862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.885237 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.885221    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.887381 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.887359    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:25.894377 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:25.894354    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4n8m\" (UniqueName: \"kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7z678\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:26.043136 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.043110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:26.173805 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.173783    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:54:26.176237 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:54:26.176210    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4179bd90_0ed8_44d7_aa20_3edb68ec49f8.slice/crio-4bf5aedacad524249f52afffeb7eb9cae2089270b056a97419245cb205169965 WatchSource:0}: Error finding container 4bf5aedacad524249f52afffeb7eb9cae2089270b056a97419245cb205169965: Status 404 returned error can't find the container with id 4bf5aedacad524249f52afffeb7eb9cae2089270b056a97419245cb205169965
Jul 08 17:54:26.768359 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.768328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" event={"ID":"4179bd90-0ed8-44d7-aa20-3edb68ec49f8","Type":"ContainerStarted","Data":"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"}
Jul 08 17:54:26.768862 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.768378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" event={"ID":"4179bd90-0ed8-44d7-aa20-3edb68ec49f8","Type":"ContainerStarted","Data":"4bf5aedacad524249f52afffeb7eb9cae2089270b056a97419245cb205169965"}
Jul 08 17:54:26.768862 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.768433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:54:26.800171 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:26.800120    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" podStartSLOduration=1.800106869 podStartE2EDuration="1.800106869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:54:26.7994363 +0000 UTC m=+4817.017632808" watchObservedRunningTime="2026-07-08 17:54:26.800106869 +0000 UTC m=+4817.018303377"
Jul 08 17:54:57.775421 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:54:57.775391    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:55:01.266944 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.266859    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:55:01.276271 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:55:01.276233    2559 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:55:01.276271 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:55:01.276271 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:55:01.276271 ip-10-0-130-182 kubenswrapper[2559]:  > containerID="58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9" cmd=["/bin/sleep","15"]
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.276283    2559 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" message="" err=<
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:  >
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:55:01.276295    2559 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:         , stdout: , stderr: , exit code -1
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerName="main" containerID="cri-o://58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"
Jul 08 17:55:01.276619 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.276371    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerName="main" containerID="cri-o://58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9" gracePeriod=60
Jul 08 17:55:01.528954 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.528930    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:55:01.678659 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.678630    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs\") pod \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") "
Jul 08 17:55:01.678811 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.678671    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds\") pod \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") "
Jul 08 17:55:01.678811 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.678704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4n8m\" (UniqueName: \"kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m\") pod \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\" (UID: \"4179bd90-0ed8-44d7-aa20-3edb68ec49f8\") "
Jul 08 17:55:01.678970 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.678944    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4179bd90-0ed8-44d7-aa20-3edb68ec49f8" (UID: "4179bd90-0ed8-44d7-aa20-3edb68ec49f8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:55:01.680954 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.680924    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m" (OuterVolumeSpecName: "kube-api-access-q4n8m") pod "4179bd90-0ed8-44d7-aa20-3edb68ec49f8" (UID: "4179bd90-0ed8-44d7-aa20-3edb68ec49f8"). InnerVolumeSpecName "kube-api-access-q4n8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:55:01.681066 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.680963    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4179bd90-0ed8-44d7-aa20-3edb68ec49f8" (UID: "4179bd90-0ed8-44d7-aa20-3edb68ec49f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:55:01.780087 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.780018    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:55:01.780087 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.780041    2559 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-tokenizer-uds\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:55:01.780087 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.780054    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4n8m\" (UniqueName: \"kubernetes.io/projected/4179bd90-0ed8-44d7-aa20-3edb68ec49f8-kube-api-access-q4n8m\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:55:01.940609 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.940558    2559 generic.go:356] "Generic (PLEG): container finished" podID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerID="58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9" exitCode=0
Jul 08 17:55:01.940779 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.940621    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"
Jul 08 17:55:01.940779 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.940651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" event={"ID":"4179bd90-0ed8-44d7-aa20-3edb68ec49f8","Type":"ContainerDied","Data":"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"}
Jul 08 17:55:01.940779 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.940708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678" event={"ID":"4179bd90-0ed8-44d7-aa20-3edb68ec49f8","Type":"ContainerDied","Data":"4bf5aedacad524249f52afffeb7eb9cae2089270b056a97419245cb205169965"}
Jul 08 17:55:01.940779 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.940735    2559 scope.go:117] "RemoveContainer" containerID="58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"
Jul 08 17:55:01.950498 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.950481    2559 scope.go:117] "RemoveContainer" containerID="58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"
Jul 08 17:55:01.950761 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:55:01.950735    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9\": container with ID starting with 58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9 not found: ID does not exist" containerID="58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"
Jul 08 17:55:01.950845 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.950772    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9"} err="failed to get container status \"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9\": rpc error: code = NotFound desc = could not find container \"58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9\": container with ID starting with 58f8c8cfa912fe26dd59d1bf8112b9043bae3fafde263220ceff8e3ba82958b9 not found: ID does not exist"
Jul 08 17:55:01.966522 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.966498    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:55:01.971064 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:01.971036    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-sched7z678"]
Jul 08 17:55:02.244737 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:02.244653    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" path="/var/lib/kubelet/pods/4179bd90-0ed8-44d7-aa20-3edb68ec49f8/volumes"
Jul 08 17:55:22.763849 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.763819    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kv6x7/must-gather-xgt5x"]
Jul 08 17:55:22.764244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.764148    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerName="main"
Jul 08 17:55:22.764244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.764158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerName="main"
Jul 08 17:55:22.764244 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.764238    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4179bd90-0ed8-44d7-aa20-3edb68ec49f8" containerName="main"
Jul 08 17:55:22.768694 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.768678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.773161 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.773140    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kv6x7\"/\"kube-root-ca.crt\""
Jul 08 17:55:22.773792 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.773772    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kv6x7\"/\"openshift-service-ca.crt\""
Jul 08 17:55:22.779535 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.779512    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kv6x7/must-gather-xgt5x"]
Jul 08 17:55:22.844214 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.844192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.844352 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.844243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrp5\" (UniqueName: \"kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.944610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.944563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrp5\" (UniqueName: \"kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.944719 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.944650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.944914 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.944901    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:22.961032 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:22.961011    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrp5\" (UniqueName: \"kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5\") pod \"must-gather-xgt5x\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") " pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:23.078733 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:23.078705    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:55:23.212702 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:23.212680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kv6x7/must-gather-xgt5x"]
Jul 08 17:55:23.214733 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:55:23.214708    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f34d2e5_f318_4daf_b980_e186a1b3b7c7.slice/crio-ec0036aaf0bd23a1446f8f1568d2875a6f58cbcf3db63cd83b239917595ce122 WatchSource:0}: Error finding container ec0036aaf0bd23a1446f8f1568d2875a6f58cbcf3db63cd83b239917595ce122: Status 404 returned error can't find the container with id ec0036aaf0bd23a1446f8f1568d2875a6f58cbcf3db63cd83b239917595ce122
Jul 08 17:55:23.216465 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:23.216450    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:55:24.054001 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:24.053936    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" event={"ID":"4f34d2e5-f318-4daf-b980-e186a1b3b7c7","Type":"ContainerStarted","Data":"ec0036aaf0bd23a1446f8f1568d2875a6f58cbcf3db63cd83b239917595ce122"}
Jul 08 17:55:30.095294 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:30.095209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" event={"ID":"4f34d2e5-f318-4daf-b980-e186a1b3b7c7","Type":"ContainerStarted","Data":"4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d"}
Jul 08 17:55:30.095294 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:30.095260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" event={"ID":"4f34d2e5-f318-4daf-b980-e186a1b3b7c7","Type":"ContainerStarted","Data":"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"}
Jul 08 17:55:30.124204 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:30.124155    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" podStartSLOduration=2.055273938 podStartE2EDuration="8.124141054s" totalImagesPullingTime="6.068867116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:55:30.122887586 +0000 UTC m=+4880.341084117" watchObservedRunningTime="2026-07-08 17:55:30.124141054 +0000 UTC m=+4880.342337562"
Jul 08 17:55:38.724404 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:38.724378    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:38.805793 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:38.805760    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:38.827019 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:38.826997    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:40.430363 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:40.430331    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:40.501057 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:40.501028    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:40.521232 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:40.521201    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:42.067270 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:42.067239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:42.123830 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:42.123791    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:42.139620 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:42.139563    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:43.698002 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:43.697977    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:43.756549 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:43.756519    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:43.771203 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:43.771179    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:45.296486 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:45.296460    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:45.359654 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:45.359630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:45.380125 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:45.380102    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:46.923190 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:46.923160    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:46.987923 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:46.987896    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:47.004536 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:47.004510    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:48.746736 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:48.746706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:48.813622 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:48.813597    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:48.831727 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:48.831695    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:50.496354 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:50.496324    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:50.567695 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:50.567657    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:50.584632 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:50.584611    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:52.204009 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:52.203982    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:52.275459 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:52.275419    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:52.290873 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:52.290851    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:53.881314 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:53.881289    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:53.940778 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:53.940749    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:53.956697 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:53.956667    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:55.511073 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:55.511042    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:55.582066 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:55.582034    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:55.597722 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:55.597690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:57.134997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:57.134958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:57.194537 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:57.194508    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:57.212793 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:57.212762    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:55:58.729705 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:58.729673    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:55:58.793841 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:58.793806    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:55:58.809729 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:55:58.809703    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:56:00.347920 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:00.347891    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:56:00.409486 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:00.409452    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:56:00.426081 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:00.426058    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:56:01.976754 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:01.976714    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:56:02.042430 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:02.042399    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:56:02.055374 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:02.055344    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:56:03.579395 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:03.579363    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-bsd5v_476696a4-2052-4220-80c5-0524c8298818/storage-initializer/0.log"
Jul 08 17:56:03.652958 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:03.652932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/main/0.log"
Jul 08 17:56:03.687745 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:03.687714    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-5c87756974-tqwvx_1eaa996a-363c-4232-abfd-bfcf8a0a1aa2/llm-d-routing-sidecar/0.log"
Jul 08 17:56:05.654900 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:05.654873    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:05.670061 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:05.670034    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:05.690361 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:05.690334    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:07.095828 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:07.095796    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:07.108692 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:07.108665    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:07.125742 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:07.125720    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:08.572585 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:08.572538    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:08.585990 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:08.585967    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:08.608681 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:08.608660    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:10.080998 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:10.080930    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:10.096199 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:10.096168    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:10.112078 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:10.112051    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:11.286784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:11.286753    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:12.740150 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:12.740123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:14.135311 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:14.135280    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:15.730847 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:15.730821    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:17.337425 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:17.337399    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:18.794784 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:18.794758    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:20.179474 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:20.179445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:21.548888 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:21.548854    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:22.936464 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:22.936440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:24.295531 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:24.295503    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:25.640233 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:25.640206    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:27.400054 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:27.400018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" event={"ID":"4f34d2e5-f318-4daf-b980-e186a1b3b7c7","Type":"ContainerDied","Data":"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"}
Jul 08 17:56:27.400054 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:27.400029    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerID="b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0" exitCode=0
Jul 08 17:56:27.400560 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:27.400355    2559 scope.go:117] "RemoveContainer" containerID="b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"
Jul 08 17:56:27.678316 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:27.678220    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kv6x7_must-gather-xgt5x_4f34d2e5-f318-4daf-b980-e186a1b3b7c7/gather/0.log"
Jul 08 17:56:31.467727 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:31.467631    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v4744_669d7b5f-cd22-4fdb-8c5a-8f11cda0533b/global-pull-secret-syncer/0.log"
Jul 08 17:56:31.578142 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:31.578111    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bbnng_5ec01533-0ac2-429d-aa0d-b016eed8f657/konnectivity-agent/0.log"
Jul 08 17:56:31.687046 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:31.687013    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-182.ec2.internal_bed7f726f53bfd3508e373fd03ee9ef0/haproxy/0.log"
Jul 08 17:56:33.332273 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.332227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kv6x7/must-gather-xgt5x"]
Jul 08 17:56:33.332828 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.332459    2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-kv6x7/must-gather-xgt5x" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="copy" containerID="cri-o://4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d" gracePeriod=2
Jul 08 17:56:33.337803 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.337779    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kv6x7/must-gather-xgt5x"]
Jul 08 17:56:33.572000 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.571979    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kv6x7_must-gather-xgt5x_4f34d2e5-f318-4daf-b980-e186a1b3b7c7/copy/0.log"
Jul 08 17:56:33.574524 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.574505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:56:33.577167 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:56:33.577142    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xgt5x\" is forbidden: User \"system:node:ip-10-0-130-182.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kv6x7\": no relationship found between node 'ip-10-0-130-182.ec2.internal' and this object" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:56:33.691617 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.691516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnrp5\" (UniqueName: \"kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5\") pod \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") "
Jul 08 17:56:33.691756 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.691628    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output\") pod \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\" (UID: \"4f34d2e5-f318-4daf-b980-e186a1b3b7c7\") "
Jul 08 17:56:33.693812 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.693783    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5" (OuterVolumeSpecName: "kube-api-access-pnrp5") pod "4f34d2e5-f318-4daf-b980-e186a1b3b7c7" (UID: "4f34d2e5-f318-4daf-b980-e186a1b3b7c7"). InnerVolumeSpecName "kube-api-access-pnrp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:56:33.698651 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.698629    2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "4f34d2e5-f318-4daf-b980-e186a1b3b7c7" (UID: "4f34d2e5-f318-4daf-b980-e186a1b3b7c7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:56:33.793283 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.793254    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnrp5\" (UniqueName: \"kubernetes.io/projected/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-kube-api-access-pnrp5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:56:33.793283 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:33.793276    2559 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f34d2e5-f318-4daf-b980-e186a1b3b7c7-must-gather-output\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\""
Jul 08 17:56:34.244651 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.244614    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" path="/var/lib/kubelet/pods/4f34d2e5-f318-4daf-b980-e186a1b3b7c7/volumes"
Jul 08 17:56:34.438431 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.438399    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kv6x7_must-gather-xgt5x_4f34d2e5-f318-4daf-b980-e186a1b3b7c7/copy/0.log"
Jul 08 17:56:34.438897 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.438761    2559 generic.go:356] "Generic (PLEG): container finished" podID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerID="4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d" exitCode=143
Jul 08 17:56:34.438897 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.438821    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kv6x7/must-gather-xgt5x"
Jul 08 17:56:34.438897 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.438874    2559 scope.go:117] "RemoveContainer" containerID="4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d"
Jul 08 17:56:34.447159 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.447139    2559 scope.go:117] "RemoveContainer" containerID="b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"
Jul 08 17:56:34.461387 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.461367    2559 scope.go:117] "RemoveContainer" containerID="4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d"
Jul 08 17:56:34.461691 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:56:34.461668    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d\": container with ID starting with 4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d not found: ID does not exist" containerID="4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d"
Jul 08 17:56:34.461793 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.461698    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d"} err="failed to get container status \"4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d\": rpc error: code = NotFound desc = could not find container \"4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d\": container with ID starting with 4c520f63276afdcd4e3b5938280bae1f66a4f73617ec452218a58d33015d5a6d not found: ID does not exist"
Jul 08 17:56:34.461793 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.461715    2559 scope.go:117] "RemoveContainer" containerID="b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"
Jul 08 17:56:34.461973 ip-10-0-130-182 kubenswrapper[2559]: E0708 17:56:34.461955    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0\": container with ID starting with b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0 not found: ID does not exist" containerID="b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"
Jul 08 17:56:34.462028 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:34.461983    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0"} err="failed to get container status \"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0\": rpc error: code = NotFound desc = could not find container \"b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0\": container with ID starting with b277685894398b5d4f9e384419cef640411884831c0b2bb4657e25fbb5628cf0 not found: ID does not exist"
Jul 08 17:56:35.559690 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:35.559664    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-28qld_987bd600-7105-452b-afdc-5f34714954fc/kuadrant-console-plugin/0.log"
Jul 08 17:56:37.802646 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:37.802610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-ktzts_95193103-a1f6-4121-a136-78b16a76b0cb/cluster-monitoring-operator/0.log"
Jul 08 17:56:38.086698 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.086610    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k6rsq_0284cda9-24a7-49c4-b37f-5ff5fb7f34fa/node-exporter/0.log"
Jul 08 17:56:38.114269 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.114245    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k6rsq_0284cda9-24a7-49c4-b37f-5ff5fb7f34fa/kube-rbac-proxy/0.log"
Jul 08 17:56:38.144736 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.144712    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k6rsq_0284cda9-24a7-49c4-b37f-5ff5fb7f34fa/init-textfile/0.log"
Jul 08 17:56:38.294223 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.294196    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-54jr6_a7197f39-e59e-45fc-9329-bff6fbaae09f/kube-rbac-proxy-main/0.log"
Jul 08 17:56:38.330262 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.330239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-54jr6_a7197f39-e59e-45fc-9329-bff6fbaae09f/kube-rbac-proxy-self/0.log"
Jul 08 17:56:38.372655 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.372566    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-54jr6_a7197f39-e59e-45fc-9329-bff6fbaae09f/openshift-state-metrics/0.log"
Jul 08 17:56:38.713321 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.713240    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kjtvx_d59253a2-e27c-4715-9306-6e7399ea8980/prometheus-operator/0.log"
Jul 08 17:56:38.744904 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.744883    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-kjtvx_d59253a2-e27c-4715-9306-6e7399ea8980/kube-rbac-proxy/0.log"
Jul 08 17:56:38.781679 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:38.781661    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-9nvfr_340a75ef-1b40-4dd0-a2af-4c1c27e49f06/prometheus-operator-admission-webhook/0.log"
Jul 08 17:56:39.000778 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.000705    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/thanos-query/0.log"
Jul 08 17:56:39.033820 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.033796    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/kube-rbac-proxy-web/0.log"
Jul 08 17:56:39.063270 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.063250    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/kube-rbac-proxy/0.log"
Jul 08 17:56:39.102869 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.102845    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/prom-label-proxy/0.log"
Jul 08 17:56:39.140425 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.140405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/kube-rbac-proxy-rules/0.log"
Jul 08 17:56:39.170818 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:39.170797    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-844c6d478b-hqmt6_9a29a01a-db28-4520-9952-20c4d1c7ad02/kube-rbac-proxy-metrics/0.log"
Jul 08 17:56:40.465713 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:40.465686    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-mt2zz_6ef29d45-9438-4c96-9e06-327eaf5fcacc/networking-console-plugin/0.log"
Jul 08 17:56:40.998010 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:40.997979    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/1.log"
Jul 08 17:56:41.002377 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:41.002354    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-vj8l6_39e92342-e9d6-4611-94fa-cd53e1d5bbd9/console-operator/2.log"
Jul 08 17:56:41.519970 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:41.519940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-v54bj_dd42600f-0b4c-49de-9456-215bcf497926/download-server/0.log"
Jul 08 17:56:42.100520 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.100477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-2kj9p_bf60097d-f13a-4efe-a406-e0d9f389f29e/volume-data-source-validator/0.log"
Jul 08 17:56:42.813618 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.813585    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"]
Jul 08 17:56:42.814027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.813949    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="gather"
Jul 08 17:56:42.814027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.813961    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="gather"
Jul 08 17:56:42.814027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.813970    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="copy"
Jul 08 17:56:42.814027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.813976    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="copy"
Jul 08 17:56:42.814162 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.814039    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="copy"
Jul 08 17:56:42.814162 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.814050    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f34d2e5-f318-4daf-b980-e186a1b3b7c7" containerName="gather"
Jul 08 17:56:42.820532 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.820510    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:42.823188 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.823157    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wf794\"/\"openshift-service-ca.crt\""
Jul 08 17:56:42.823334 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.823167    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wf794\"/\"kube-root-ca.crt\""
Jul 08 17:56:42.824371 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.824356    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wf794\"/\"default-dockercfg-97555\""
Jul 08 17:56:42.829183 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.829161    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"]
Jul 08 17:56:42.983404 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.983369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrd9l\" (UniqueName: \"kubernetes.io/projected/91850ce6-c449-4574-9095-946feaeda2af-kube-api-access-lrd9l\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:42.983610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.983445    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-lib-modules\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:42.983610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.983476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-proc\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:42.983610 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.983564    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-sys\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:42.983730 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:42.983620    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-podres\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.084887 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-sys\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.084887 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-podres\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.084887 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrd9l\" (UniqueName: \"kubernetes.io/projected/91850ce6-c449-4574-9095-946feaeda2af-kube-api-access-lrd9l\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-lib-modules\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084934    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-sys\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084943    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-proc\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.084991    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-podres\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.085002    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-proc\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.085121 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.085052    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91850ce6-c449-4574-9095-946feaeda2af-lib-modules\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.094740 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.094709    2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrd9l\" (UniqueName: \"kubernetes.io/projected/91850ce6-c449-4574-9095-946feaeda2af-kube-api-access-lrd9l\") pod \"perf-node-gather-daemonset-lsvj2\" (UID: \"91850ce6-c449-4574-9095-946feaeda2af\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.130713 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.130689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.202139 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.202106    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gglb5_ebea962f-1221-4e2c-8479-5bd75507fe4b/dns/0.log"
Jul 08 17:56:43.228997 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.228972    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gglb5_ebea962f-1221-4e2c-8479-5bd75507fe4b/kube-rbac-proxy/0.log"
Jul 08 17:56:43.269858 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.269835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"]
Jul 08 17:56:43.272260 ip-10-0-130-182 kubenswrapper[2559]: W0708 17:56:43.272230    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod91850ce6_c449_4574_9095_946feaeda2af.slice/crio-2b1762e878fd5c461ffe5c3bf3b662f06d29cc95cc09ad7cb370f2088cadc703 WatchSource:0}: Error finding container 2b1762e878fd5c461ffe5c3bf3b662f06d29cc95cc09ad7cb370f2088cadc703: Status 404 returned error can't find the container with id 2b1762e878fd5c461ffe5c3bf3b662f06d29cc95cc09ad7cb370f2088cadc703
Jul 08 17:56:43.376695 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.376671    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d558v_25359783-c9a1-4d17-b19d-cd028d55d434/dns-node-resolver/0.log"
Jul 08 17:56:43.489011 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.488982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2" event={"ID":"91850ce6-c449-4574-9095-946feaeda2af","Type":"ContainerStarted","Data":"36d9dd44f904e7d057356de8d8e0e0f0212cd82981da84ba3bccc6bcfefe829a"}
Jul 08 17:56:43.489205 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.489028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2" event={"ID":"91850ce6-c449-4574-9095-946feaeda2af","Type":"ContainerStarted","Data":"2b1762e878fd5c461ffe5c3bf3b662f06d29cc95cc09ad7cb370f2088cadc703"}
Jul 08 17:56:43.489205 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.489103    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:43.508117 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:43.508056    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2" podStartSLOduration=1.508039942 podStartE2EDuration="1.508039942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:56:43.507551753 +0000 UTC m=+4953.725748262" watchObservedRunningTime="2026-07-08 17:56:43.508039942 +0000 UTC m=+4953.726236450"
Jul 08 17:56:44.074288 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:44.074252    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mdplr_797e5d6b-c7bb-4df7-a1c8-297eca0fa2e1/node-ca/0.log"
Jul 08 17:56:45.590747 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:45.590706    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:45.603771 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:45.603743    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:45.633027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:45.633000    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:46.179380 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:46.179348    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9sr8f_3d7d00a0-116b-44f1-94b0-1d5a053a99a0/serve-healthcheck-canary/0.log"
Jul 08 17:56:46.743760 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:46.743729    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jcfjt_7027dd97-8f1d-4c79-9429-8f9b6309bae3/insights-operator/0.log"
Jul 08 17:56:46.744320 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:46.744304    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-jcfjt_7027dd97-8f1d-4c79-9429-8f9b6309bae3/insights-operator/1.log"
Jul 08 17:56:46.951036 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:46.951009    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9d9h_ffa362c9-df89-4ef1-86f3-464642a161f4/kube-rbac-proxy/0.log"
Jul 08 17:56:46.983269 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:46.983242    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9d9h_ffa362c9-df89-4ef1-86f3-464642a161f4/exporter/0.log"
Jul 08 17:56:47.013712 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:47.013634    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9d9h_ffa362c9-df89-4ef1-86f3-464642a161f4/extractor/0.log"
Jul 08 17:56:49.504438 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:49.504412    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-lsvj2"
Jul 08 17:56:49.898530 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:49.898497    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6744679755-9gjmt_8929e446-21c0-4a05-8e57-e085f97b3880/manager/0.log"
Jul 08 17:56:50.594277 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:50.594245    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-76d8d77597-mztr2_b31bc5c2-26be-45f6-8215-cbdf59ae0d4c/manager/0.log"
Jul 08 17:56:51.564717 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:51.564692    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mc5rq_fdd86e2b-679c-4dba-9e80-9d53809c0b5e/s3-init/0.log"
Jul 08 17:56:51.583677 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:51.583652    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mc5rq_fdd86e2b-679c-4dba-9e80-9d53809c0b5e/download-hf-model/0.log"
Jul 08 17:56:51.614875 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:51.614854    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-jkjfv_fd50a91b-0191-4bfe-8c64-b29c35834960/seaweedfs/0.log"
Jul 08 17:56:52.250467 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:52.250440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:52.265020 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:52.264997    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:52.278845 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:52.278825    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:53.838219 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:53.838186    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/0.log"
Jul 08 17:56:53.850910 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:53.850892    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-mbx7z_1fb658bd-2735-49af-a894-31ceab2c12ac/istio-proxy/1.log"
Jul 08 17:56:53.873539 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:53.873516    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5658d689b9-4zvzx_7347fcff-b97d-48df-a5a1-b39adcae0f7b/router/0.log"
Jul 08 17:56:57.573671 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:57.573612    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-6c6v5_610465db-6b6f-4151-ba7a-08a61e6ede7c/kube-storage-version-migrator-operator/1.log"
Jul 08 17:56:57.574872 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:57.574854    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-6c6v5_610465db-6b6f-4151-ba7a-08a61e6ede7c/kube-storage-version-migrator-operator/0.log"
Jul 08 17:56:58.667046 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:58.667014    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8pv6c_7f231f0b-d60d-4606-a528-5c77859ace84/kube-multus/0.log"
Jul 08 17:56:59.060621 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.060567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:56:59.087366 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.087331    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/egress-router-binary-copy/0.log"
Jul 08 17:56:59.113852 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.113829    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/cni-plugins/0.log"
Jul 08 17:56:59.143155 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.143121    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/bond-cni-plugin/0.log"
Jul 08 17:56:59.169691 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.169669    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/routeoverride-cni/0.log"
Jul 08 17:56:59.193177 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.193153    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/whereabouts-cni-bincopy/0.log"
Jul 08 17:56:59.219028 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.219004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stzpv_81401226-bc48-426d-878e-bca23eb723e4/whereabouts-cni/0.log"
Jul 08 17:56:59.354413 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.354348    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nb49f_26860848-e85c-4136-9004-bb2c83bfe483/network-metrics-daemon/0.log"
Jul 08 17:56:59.388966 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:56:59.388940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nb49f_26860848-e85c-4136-9004-bb2c83bfe483/kube-rbac-proxy/0.log"
Jul 08 17:57:00.930066 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:00.929978    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/ovn-controller/0.log"
Jul 08 17:57:00.950277 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:00.950236    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/ovn-acl-logging/0.log"
Jul 08 17:57:00.977963 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:00.977936    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/kube-rbac-proxy-node/0.log"
Jul 08 17:57:01.001940 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:01.001919    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:57:01.024027 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:01.024006    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/northd/0.log"
Jul 08 17:57:01.047485 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:01.047462    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/nbdb/0.log"
Jul 08 17:57:01.070449 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:01.070411    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/sbdb/0.log"
Jul 08 17:57:01.126281 ip-10-0-130-182 kubenswrapper[2559]: I0708 17:57:01.126255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rqpfq_6d2a3089-f878-4543-8cec-c7dc83e9c457/ovnkube-controller/0.log"